Error if running zero tests

[RubenVerborgh]

In SPARQL.js, running npm run spec-base-query now silently fails with ”0/0 tests”:

> [email protected] spec-base-query
> rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest.ttl -s http://www.w3.org/TR/sparql11-query/ -c .rdf-test-suite-cache/

Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv01.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv02.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv03.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv01.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv02.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv03.tsv
✔ 0 / 0 tests succeeded!

[rubensworks]

In comunica we use http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl, could you try that one?

I'm guessing some breaking changes were introduced in http://w3c.github.io/rdf-tests/sparql/sparql11/manifest.ttl for the RDF 1.2 tests.

[RubenVerborgh]

Thanks! That's indeed quite confusing, especially given that they broke URIs repeatedly (RubenVerborgh/SPARQL.js@0ef0d4c).

With npx rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl -s http://www.w3.org/TR/sparql11-query/ -c .rdf-test-suite-cache/, I get: ✔ 301 / 301 tests succeeded! (skipped 198). The 198 still seems like a lot, but perhaps that's okay?

For npx rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-sparql11-update.ttl -s http://www.w3.org/TR/sparql11-update/ -c .rdf-test-suite-cache/, I unfortunately still have ✔ 0 / 0 tests succeeded!.

Suggestion: still error on 0/0?

[rubensworks]

I get: ✔ 301 / 301 tests succeeded! (skipped 198). The 198 still seems like a lot, but perhaps that's okay?

That should be fine indeed, since the passed runner is configured to skip all non-parsing tests: https://github.com/RubenVerborgh/SPARQL.js/blob/main/spec/parser.js#L4-L15

For npx rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-sparql11-update.ttl -s http://www.w3.org/TR/sparql11-update/ -c .rdf-test-suite-cache/, I unfortunately still have ✔ 0 / 0 tests succeeded!.

http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl should also work here.
At least it does in comunica: https://github.com/comunica/comunica/blob/master/engines/query-sparql/package.json#L54-L56

[RubenVerborgh]

Thanks, LGTM. Maybe just consider if you want the suite to error if it finds 0 tests.

[rubensworks]

Maybe just consider if you want the suite to error if it finds 0 tests.

Good idea, we should add that.

In addition, we may also want to add a flag that does not error in such cases, as there may be cases where zero tests are valid.