How to pass temp dir to tdbloader2

[marksmall]

Hi there,

I'm trying to run the tdbloader2 command but it consistently fails during sorting, more annoyingly, it seems to fail silently. I have got it to work with a small number of files, about 300, but when I went up to 1000 (true total is 2048), then it successfully passes the Load Data Phase, but during the Build Index Phase it quitely fails on SPO. I expect to see something like Completed index build SPO and move on to the other indexes, but what I actually see is nothing, it starts indexing, obviously fails and quietly moves on to the next steps in my script. I have tried to run this manually, but the same thing, it just quietly fails and exists. I have tried adding --debug and that didn't give me any indication as to the problem either, so I'm looking for some advice.

It is running inside kubernetes with 85G free diskspace and allowed up to 16G of memory. I am have tried to set the --sort-args=--temporary-directory=/var/lib/tmp which is a mounted disk with GBs of diskspace, as I have experienced this problem before, on my local machine, but was able to clear out disk space so it was over 100Gb, but I cannot do that on the kubernetes pod, I'm restricted to what is available, which is why I'm trying to set the temporary-directory setting.

Any advice would be greatly appreciated, thanks in advance.

[rvesse]

--sort-args=--temporary-directory=/var/lib/tmp should be a valid option to the script AFAICT

If you are running with --debug then you should see some messages like the following:

Sort Temp Directory is on disk ${SORT_DRIVE_INFO[0]} which has ${SORT_DRIVE_INFO[2]}% free space (${SORT_DRIVE_INFO[3]} bytes)

Obviously those variables would be substituted with the real values in your environment

You mention Kubernetes so the one thing that comes to mind is how are you invoking your script? Are you calling it from an entrypoint script in your container image, manually from a shell, via the command and/or args portions of your pod spec?

Whichever of those it is I would make sure you are properly escaping the arguments because it's possible that --sort-args=--temporary-directory=var/lib/tmp is getting misinterpreted and split into multiple arguments that would result in it not being honoured

Note that the script honours the value of the TMPDIR environment variable if you haven't tried to pass it in via --sort-args so an alternative is just to set that to the desired directory before invoking the script

In general it would be useful to confirm exactly which version of Jena you are using and how you invoke the tdbloader2 script. In recent versions of Jena that script actually just prints a warning that it has been renamed to something else so if you're able to run that script directly it implies you may be using an outdated version of Jena. Thus another possibility is that you're simply using a version of Jena that does not support configuring the temporary directory.

[marksmall]

Thanks for such a prompt reply @rvesse, your right, I should have stated the version etc.

The version I am using is 3.4.0, the reason is this is the version currently being used by the LIVE app I am migrating, so I didn't want to make big changes without a good reason. I think though it might be time to try it with the latest release and amend the scripts to use the correct binary, so it is easier to get help, if further help would be needed.

It's good to know it should honour the TMPDIR setting, so I can set that and try to avoid setting the dir to use in the script. When I have everything working, the script will be invoked from the entrypoint.sh script I have for my docker container, but right now I'm exec'ing into the pod and manually running it, so I can bypass some of the more time-consuming aspects, e.g. converting 2000 CSV files to TTL files.

[rvesse]

AFAICT 3.4.0 does support both --sort-args and honours the TMPDIR setting for sort if not set via --sort-args so you should be able to use that version if you need to

[afs]

3.4.0

I'm note sure what's in 3.4.0 . There are important security upgrades and fixes.

tdbloader2 has been renamed tdb1.xloader. There was a lot of work on improving it. But are you sure you need that need the that loader and not tdb1.tdbloader?

tdb1.xloader from a later version should produce a compatible database.

More work has gone into the TDB2 loader. As well as a xloader style loader, there is a multithreaded loader which is easier to use.

[marksmall]

This is all great stuff thanks, gives me avenues to investigate. If I were to upgrade, we use both tdbloader2 which now seems to be tdb1.xloader, but we also use tdbloader. The reason, as I understand from the old maintainers of the app are:

• tdbloader2 - is used to generate a brand new database once per month.
• tdbloader - is used to append daily data to the existing database.

If I were to upgrade, what would be the tdbloader equivalent?

[marksmall]

Some further background. The commands all work fine when I use say 300 files, but fails when I use 1000 files, meaning more files to work with and more data, the total will be closer to 2000+ files. It consistently fails during the sorting, so it can only be memory or disk-space, if you disagree though, please just say. The actual error I see is:

/opt/apache-jena-3.4.0/bin/tdbloader2index: line 306: 109143 Killed                  sort $SORT_ARGS -u $KEYS < "$DATA" > $WORK                                                                                    

The logging I'm seeing is:

12:09:41 DEBUG JVM Arguments are -Xmx1200M
 12:09:41 DEBUG Jena Classpath is /opt/apache-jena-3.4.0/lib/*
 12:09:41 INFO Index Building Phase
 12:09:41 DEBUG Sort Arguments: --buffer-size=50% --parallel=3
 12:09:41 DEBUG Sort Temp Directory: /var/lib/wims-staging/
 12:09:41 DEBUG Sort Temp Directory is on disk //fac25d2a92a4740cf8686bc.file.core.windows.net/pvc-69ec4bea-4244-4249-aa61-a27910cbc6a7 which has 70% free space (113054253056 bytes)
 12:09:41 INFO Creating Index SPO
 12:09:41 DEBUG Size of data to be sorted is 8902588356 bytes
 12:09:41 DEBUG Sufficient free space on database drive //fac25d2a92a4740cf8686bc.file.core.windows.net/pvc-ade1ddd5-a53f-421b-92b2-cf81a2144b94 to attempt sorting data file /fuseki/databases/green/DS-DB//data-triples.tmp (8902588356 bytes required from 201771515904 bytes free)
 12:09:41 DEBUG Should be sufficient free memory (14374588416 bytes) for sort to be fully in-memory
 12:09:41 INFO Sort SPO
 12:09:41 DEBUG Sorting /fuseki/databases/green/DS-DB//data-triples.tmp into work file /fuseki/databases/green/DS-DB//SPO-txt

Suggesting it can do it in memory. Since the error is in the tdbloader2index script, which I don't directly interact with, but is called by tdbloader2. I'm not sure what else I can do, other than upgrade the version of fuseki being used.

[rvesse]

I am wondering if you're running into the OOM killer inside the container. sort runs and tries to hold all the data in memory per:

-S size, --buffer-size=size
Use size for the maximum size of the memory buffer. Size modifiers %,b,K,M,G,T,P,E,Z,Y can be used. If
a memory limit is not explicitly specified, sort takes up to about 90% of available memory. If the file
size is too big to fit into the memory buffer, the temporary disk files are used to perform the sorting.

Which depending on the container runtime might compete enough with other processes inside the container to have the OOM killer kill it off.

For running in a container with constrained memory you might need to add -S 1G (or some other low value) to --sort-args to force sort to use a disk based sort instead.

[afs]

Should be sufficient free memory (14374588416 bytes) for sort to be fully in-memory

That's approaching 14G.
It's a 16G machine, the Java heap is 1.2G. Looks close.

I think --buffer-size=50% --parallel=3 means 3 buffers each of a suggested 50% of free RAM.

[marksmall]

Thanks guys, I did the first suggestion of setting it to 1G and it has worked, so maybe I just have to force disk-based sorting. I'll try a couple of times though just in case, I will also try the options @afs added as well. Definitely heading in the right direction though, thanks for all the help.