Maven package command fills hard drive with temporary test files

[whereswaldon]

On OS X El Capitan, running mvn package runs the nashorn tests by default. These tests create millions of temporary files that slowly consume the host computer's hard drive. The first time that I did this, it create 440GB of temporary files. I assumed that I had done something wrong and tried again today, but I cut it short after about 50GB of temporary files. They are deleted when the system reboots, but it's still a problem if a user can't run the test suite.

I don't know what the files are for, but I'm guessing that they're created as part of the testing process. If that's the case, can they be deleted in the teardown phase of each test? (If this is wildly off-base because I don't know anything about the test framework, please forgive me)

[billreed63]

Where are these tmp files created, I have OS X El Capitan and have never seen this. How long does your mvn package take to run?

[whereswaldon]

I just updated to master and ran mvn package again. It ran for around five minutes before I killed it. In that time, it populated /private/var/folders/l9/3b7w9lpj7w76_rwbhrvp62r00000gn/T/blockmgr-e253e322-965b-4f2d-be58-4d23688779f5/with a bunch of folders with two-digit hexadecimal names like 14 or 0e. Each of these directories is filled with files with names like shuffle_43_9389_0.index. During the five minutes that I ran it, it created over 8GB of content within the above directory. I once tried to let the tests finish, which resulted in (an hour later) a warning that my computer was running out of hard drive space. During that run, it created 440GB.

[Brian-Burns-Bose]

This seems like a memory issue, spark is persisting to disk because it's
out of memory. I wonder if forking the maven processes would work better?
Maybe we are in a situation where maven and spark are hogging all the JVM
memory for the process. There has to be a way to tweak this.

On Thu, Jun 23, 2016 at 7:37 AM, Christopher Waldon <
[email protected]> wrote:

I just updated to master and ran mvn package again. It ran for around
five minutes before I killed it. In that time, it populated
/private/var/folders/l9/3b7w9lpj7w76_rwbhrvp62r00000gn/T/blockmgr-e253e322-965b-4f2d-be58-4d23688779f5/with
a bunch of folders with two-digit hexadecimal names like 14 or 0e. Each
of these directories is filled with files with names like
shuffle_43_9389_0.index. During the five minutes that I ran it, it
created over 8GB of content within the above directory. I once tried to let
the tests finish, which resulted in (an hour later) a warning that my
computer was running out of hard drive space. During that run, it created
440GB.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#277 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAiWKJRSsSbiY8p5Su50YpOcpNK64sC7ks5qOnAHgaJpZM4I6sB4
.

[Brian-Burns-Bose]

I am seeing the same thing. I killed mvn package after 10 minutes and it
didn't get very far.

On Thu, Jun 23, 2016 at 8:01 AM, Brian Burns [email protected]
wrote:

This seems like a memory issue, spark is persisting to disk because it's
out of memory. I wonder if forking the maven processes would work better?
Maybe we are in a situation where maven and spark are hogging all the JVM
memory for the process. There has to be a way to tweak this.

On Thu, Jun 23, 2016 at 7:37 AM, Christopher Waldon <
[email protected]> wrote:

I just updated to master and ran mvn package again. It ran for around
five minutes before I killed it. In that time, it populated
/private/var/folders/l9/3b7w9lpj7w76_rwbhrvp62r00000gn/T/blockmgr-e253e322-965b-4f2d-be58-4d23688779f5/with
a bunch of folders with two-digit hexadecimal names like 14 or 0e. Each
of these directories is filled with files with names like
shuffle_43_9389_0.index. During the five minutes that I ran it, it
created over 8GB of content within the above directory. I once tried to let
the tests finish, which resulted in (an hour later) a warning that my
computer was running out of hard drive space. During that run, it created
440GB.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#277 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAiWKJRSsSbiY8p5Su50YpOcpNK64sC7ks5qOnAHgaJpZM4I6sB4
.

[billreed63]

I just pulled the latest master and did

mvn clean
mvn package

I get a clean build and run of the test cases

[INFO] Building jar: /Users/billreed/eclairjs_dev/eclairjs-nashorn/target/eclairjs-nashorn-0.6-SNAPSHOT-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:48 min
[INFO] Finished at: 2016-06-23T09:52:49-04:00
[INFO] Final Memory: 80M/619M
[INFO] ------------------------------------------------------------------------

mvn --version
MacBook-Pro:eclairjs-nashorn billreed$ mvn --version
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T07:57:37-04:00)
Maven home: /usr/local/Cellar/maven/3.3.3/libexec
Java version: 1.8.0_91, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: "mac"
MacBook-Pro:eclairjs-nashorn billreed$ 

[Brian-Burns-Bose]

It works fine on my mac pro with 32G ram. On my mac book with 16G it hangs. Both machines are on maven 3.3.9.

[whereswaldon]

I'm running

$ mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_92, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: "mac"

on a Macbook Pro Retina from Early 2013 with 16GB of RAM and a 2.8GHz i7.

[billreed63]

I have the same specs as @whereswaldon so it is not memory I would bet it is mvn I am running Apache Maven 3.3.3 both @bpburns and @whereswaldon are running 3.3.9