StreamRDF RIOT and generalized RDF

[Ostrzyciel]

I've been on a witch hunt for generalized RDF-related code recently. Generalized statements typically are used only within one application (e.g., as a result of RDFS reasoning), and are rarely exchanged in serialized form. When the reasoner output is piped to an RDF writer, we get issues like the one recently discussed on the user list: https://lists.apache.org/thread/vgw50ztx8vtz4g57ybcrykpndplmngcg

I was wondering what RDF readers/writers actually support generalized RDF. I did a quick experiment and it turns out that (detailed results in collapsed section below):

• Turtle "blocks" format fails on writing. In my other tests it also failed on reading.
• N-Triples writes fine, but fails on reading.
• Thrift and Protobuf binary formats write and read just fine.

Results

N-Triples/utf-8
org.apache.jena.riot.RiotException: [line: 1, col: 1 ] Expected BNode or IRI: Got: [STRING:subject]
    at org.apache.jena.riot.system.ErrorHandlerFactory$ErrorHandlerStd.fatal(ErrorHandlerFactory.java:155)
    at org.apache.jena.riot.lang.LangEngine.raiseException(LangEngine.java:148)
    at org.apache.jena.riot.lang.LangEngine.exceptionDirect(LangEngine.java:143)
    at org.apache.jena.riot.lang.LangEngine.exception(LangEngine.java:137)
    at org.apache.jena.riot.lang.LangNTuple.checkIRIOrBNode(LangNTuple.java:128)
    at org.apache.jena.riot.lang.LangNTuple.parseTriple(LangNTuple.java:91)
    at org.apache.jena.riot.lang.LangNTriples.parseOne(LangNTriples.java:61)
    at org.apache.jena.riot.lang.LangNTriples.runParser(LangNTriples.java:53)
    at org.apache.jena.riot.lang.LangBase.parse(LangBase.java:43)
    at org.apache.jena.riot.lang.RiotParsers$AbstractReaderRIOTLang.read(RiotParsers.java:133)
    at org.apache.jena.riot.lang.RiotParsers$AbstractReaderRIOTLang.read(RiotParsers.java:91)
    at org.apache.jena.riot.RDFParser.read(RDFParser.java:453)
    at org.apache.jena.riot.RDFParser.parseNotUri(RDFParser.java:443)
    at org.apache.jena.riot.RDFParser.parse(RDFParser.java:380)
    at org.apache.jena.riot.RDFParserBuilder.parse(RDFParserBuilder.java:552)
    at org.example.Main.main(Main.java:46)
java.lang.UnsupportedOperationException: "predicate" is not a URI node
    at org.apache.jena.graph.Node.getURI(Node.java:211)
    at org.apache.jena.riot.writer.Widths.calcWidth(Widths.java:36)
    at org.apache.jena.riot.writer.Widths.calcWidthTriples(Widths.java:71)
    at org.apache.jena.riot.writer.WriterStreamRDFBlocks.writePredicateObjectList(WriterStreamRDFBlocks.java:143)
    at org.apache.jena.riot.writer.WriterStreamRDFBlocks.printBatch(WriterStreamRDFBlocks.java:135)
    at org.apache.jena.riot.writer.WriterStreamRDFBlocks.printBatchTriples(WriterStreamRDFBlocks.java:107)
    at org.apache.jena.riot.writer.WriterStreamRDFBatched.finishBatchTriples(WriterStreamRDFBatched.java:100)
    at org.apache.jena.riot.writer.WriterStreamRDFBatched.flush(WriterStreamRDFBatched.java:59)
    at org.apache.jena.riot.writer.WriterStreamRDFBatched.endData(WriterStreamRDFBatched.java:56)
    at org.apache.jena.riot.writer.WriterStreamRDFBase.finish(WriterStreamRDFBase.java:125)
    at org.apache.jena.riot.system.StreamRDFWrapper.finish(StreamRDFWrapper.java:57)
    at org.example.Main.main(Main.java:41)
Writer: OK
Reader: FAIL


Turtle/blocks
Writer: FAIL
Reader: FAIL


RDF-PROTO
"subject" "predicate" "object" .
Writer: OK
Reader: OK


RDF-THRIFT
"subject" "predicate" "object" .
Writer: OK
Reader: OK

For me the weirdest part here is the N-Triples RIOT that can't read what it just wrote 😆

My angle is that I'm developing a format (Jelly) that has a file header telling the reader whether generalized RDF may lie inside the file. It would be nice to set this property to "false" in situations where generalized RDF can't be produced. Maybe even separate somehow the RIOTs that can do generalized RDF and those that can't.

From what I understand, it's impossible to put generalized RDF in Model and Dataset, but it can be present in Graph and DatasetGraph. So, if we are serializing a Model/Dataset, we can safely assume that no generalized RDF will be there.

For StreamRDF I know that there is a filter of generalized RDF that can be applied, but it's impossible to tell for a downstream consumer if this filter was applied. So, we don't know if there will be generalized RDF in the stream or not.

StreamRDF instances generated from the new Titanium RDF API will also never contain generalized RDF, because Titanium doesn't support that: #3061

Questions from my side, up for discussion:

• Should there be a way for StreamRDF producers to inform the consumers that there may/may not be generalized RDF in the stream?
• Should RIOT implementations expose an API telling whether they support generalized RDF or not?
• Should the N-Triples RIOT support only writing of generalized RDF (like right now), reading and writing, or neither?

...feel free to add anything else that comes to your mind, I'm just doing a "thought dump" here.

[afs]

• Should there be a way for StreamRDF producers to inform the consumers that there may/may not be generalized RDF in the stream?

StreamRDF is a general low level plumbing, to plug RDF processing together, focused on parser output. It separates producer and consumer implementation choices. It is not arbitrary plug-and-play though. For example, e.g. in many cases, there is no quad support.

To provide context/options, I would suggest a layer on top of StreamRDF that focuses on being a processing framework.

• Should RIOT implementations expose an API telling whether they support generalized RDF or not?

Jena is not offering a generalized RDF processing platform. The assumptions of proper RDF runs deep.

("generalized" is particularly hard because the predicate position is general - e.g. triple terms).

• Should the N-Triples RIOT support only writing of generalized RDF (like right now), reading and writing, or neither?

The parsers are RDF, not generalized RDF. Uses expect errors on "bad RDF".

There is no reason why new languages can't be added but redefining standard languages would be bad.

[Ostrzyciel]

Thanks! That makes a lot of sense.

From my perspective, the most useful thing here would be to have this layer on top of StreamRDF that you suggested. Perhaps as an interface extending StreamRDF? For me just a WellFormedStreamRDF would be enough, but a more general interface exposing and propagating options would work fine as well.