blocking bash pipelines

[chiarcos]

This seems to happen in the following pipeline (cf. https://github.com/acoli-repo/germhist, analyses/scrambling/Makefile, slightly modified version of make chlg.tsv: added tee commands)

cat $file | \
# split into columns WORD POS PARSE \
perl -pe 's/^(.*)\(([^\(\) ]+) +([^\(\)]+)\)/\3\t\2\t\1*/;' | \
# convert to CoNLL-RDF with tree extensions \
tee chlg.1.tmp| \
conll-rdf/run.sh CoNLLBrackets2RDF file:`realpath $file`'#' WORD POS PARSE | \
tee chlg.2.ttl| \
# extrapolate edge labels \
conll-rdf/run.sh CoNLLRDFUpdater -custom -updates enhgprep.sparql |\
tee chlg.3.ttl |\
    conll-rdf/run.sh CoNLLRDFFormatter -query enhgquery.sparql | \
    egrep '.' | grep -v '^#' | \
    sed s/'$$'/'\t'$$c'\t'$$h'\t'$$genre'\t'$$region/|\
    tee chlg.tsv

1. When tracking the progress of the script with ls -l chlg*, it seems that chlg3.ttl remains empty until chlg2.ttl is no longer written into. Maybe, CoNLLBrackets2RDF doesn't produce the right sentence delimiter symbol.
2. The same thing seems to happen between writing chlg.3.ttl and chlg.tsv (also checked with ls -l *).

[chiarcos]

I can confirm that CoNLLRDFUpdater doesn't work in streaming mode. Looking at chldg.3.ttl, I can indeed confirm that this is processed en block, but not streamed: chldg.3.ttl is not grouped by sentences, but has random order from all over the the input sentence data.

[chiarcos]

I guess this is because the CoNLLBrackets2RDF output uses the "new" SPARQL prefixes, but

conll-rdf/src/main/java/org/acoli/conll/rdf/CoNLLRDFUpdater.java

Line 812 in 15a3d20

if(!buffer.trim().equals("") && (line.startsWith("@") || line.startsWith("#")) && !lastLine.startsWith("@") && !lastLine.startsWith("#")) { //!buffer.matches("@[^\n]*\n?$")) {

only uses the old ones for splitting and namespace spotting.

[chiarcos]

so, this is actually the same issue as #80