Usage of Jena 5.2.0 in a native-image build failing

[jerdebsp]

Hi all,

I am using Jena as part of a Spring boot application that I am trying to build as a native image using graalvm. Whilst the whole process works, Jena seems to be failing, mainly the Jena system and subsystems are not being loaded - see exception below:

Exception in thread "main" java.lang.IllegalStateException: java.lang.ExceptionInInitializerError
        at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:825)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:345)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352)
        at com.sp.Application.main(Application.java:15)
        at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)
Caused by: java.lang.ExceptionInInitializerError
        at org.apache.jena.ext.xerces.impl.dv.SchemaDVFactory.<clinit>(SchemaDVFactory.java:45)
        at org.apache.jena.datatypes.xsd.XSDDatatype.<init>(XSDDatatype.java:234)
        at org.apache.jena.datatypes.xsd.XSDDatatype.<init>(XSDDatatype.java:246)
        at org.apache.jena.datatypes.xsd.impl.XSDFloat.<init>(XSDFloat.java:49)
        at org.apache.jena.datatypes.xsd.XSDDatatype.<clinit>(XSDDatatype.java:54)
        at org.apache.jena.sparql.function.StandardFunctions.loadStdDefs(StandardFunctions.java:69)
        at org.apache.jena.sparql.function.FunctionRegistry.init(FunctionRegistry.java:50)
        at org.apache.jena.query.ARQ.init(ARQ.java:665)
        at org.apache.jena.sparql.system.InitARQ.start(InitARQ.java:29)
        at org.apache.jena.base.module.Subsystem.lambda$initialize$1(Subsystem.java:117)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1597)
        at org.apache.jena.base.module.Subsystem.forEach(Subsystem.java:193)
        at org.apache.jena.base.module.Subsystem.forEach(Subsystem.java:169)
        at org.apache.jena.base.module.Subsystem.initialize(Subsystem.java:115)
        at org.apache.jena.sys.JenaSystem.init(JenaSystem.java:90)
        at org.apache.jena.sparql.core.DatasetGraphFactory.<clinit>(DatasetGraphFactory.java:34)
        at org.apache.jena.query.DatasetFactory.createTxnMem(DatasetFactory.java:69)
        at com.sp.Processor.<init>(Processor.java:49)
        at com.sp.Processor.getInstance(Processor.java:54)
        at com.sp.shell.commands.EngineCommand.init(EngineCommand.java:72)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
        at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
        at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:230)
        at org.springframework.shell.Shell.evaluate(Shell.java:248)
        at org.springframework.shell.Shell.run(Shell.java:159)
        at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:72)
        at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:66)
        at org.springframework.boot.SpringApplication.lambda$callRunner$4(SpringApplication.java:786)
        at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83)
        at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60)
        at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:786)
        at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774)
        at [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at [email protected]/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
        at [email protected]/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:557)
        at [email protected]/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
        at [email protected]/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at [email protected]/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
        at [email protected]/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:342)
        ... 4 more
Caused by: java.lang.RuntimeException: internal error
        at org.apache.jena.ext.xerces.impl.dv.xs.XSSimpleTypeDecl.applyFacets1(XSSimpleTypeDecl.java:768)
        at org.apache.jena.ext.xerces.impl.dv.xs.BaseSchemaDVFactory.createBuiltInTypes(BaseSchemaDVFactory.java:205)
        at org.apache.jena.ext.xerces.impl.dv.xs.SchemaDVFactoryImpl.createBuiltInTypes(SchemaDVFactoryImpl.java:44)
        at org.apache.jena.ext.xerces.impl.dv.xs.SchemaDVFactoryImpl.<clinit>(SchemaDVFactoryImpl.java:39)
        ... 49 more

Just to give some more context, the application is loading, however hitting this error first time I am creating an instance and calling DatasetFactory.createTxnMem(); in the constructor.

I am using the Jena POM dependency and also create and use the shaded JAR in the packaging:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <id>shade-jar-with-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <shadedArtifactAttached>true</shadedArtifactAttached>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.handlers</resource>
                                </transformer>
                                <transformer
                                        implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                                    <resource>META-INF/spring.factories</resource>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.schemas</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.sp.Application</mainClass>
                                    <manifestEntries>
                                        <Multi-Release>true</Multi-Release>
                                    </manifestEntries>
                                </transformer>
                            </transformers>

                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <!-- Some jars are signed but shading breaks that. Don't
                                          include signing files. -->
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                        <exclude>META-INF/DEPENDENCIES</exclude>
                                        <exclude>META-INF/MANIFEST.MF</exclude>
                                        <exclude>**/module-info.class</exclude>
                                    </excludes>
                                </filter>
                            </filters>

                        </configuration>
                    </execution>
                </executions>
            </plugin>
....
            <plugin>
                <groupId>org.graalvm.buildtools</groupId>
                <artifactId>native-maven-plugin</artifactId>
                <extensions>true</extensions>
                <executions>
                    <execution>
                        <id>build-native</id>
                        <goals>
                            <goal>build</goal>
                        </goals>
                        <phase>package</phase>
                    </execution>
                </executions>
                <configuration>
                    <skip>false</skip>
                    <useArgFile>true</useArgFile>
                    <fallback>false</fallback>
                    <classpath>
                        <param>${project.build.directory}/${project.artifactId}-${project.version}-shaded.jar</param>
                    </classpath>
                </configuration>
            </plugin>

I've tried various things, including using the direct dependencies instead of the Jena POM, but nothing seemed to work. Any potential ideas what i might be missing?

Thanks
Jer

[afs]

It looks like the service loader setup has been broken.

• Does the shared jar work before making into a native image?
• In the shared jar, do the service/ files appear and look right? (There are several transforms and there may be an interaction between them).

I found mention of -H:+TraceServiceLoaderFeature to debug in Graalvm.

[jerdebsp]

Hi Andy,

Does the shared jar work before making into a native image?

Yes it does.

In the shared jar, do the service/ files appear and look right? (There are several transforms and there may be an interaction between them).

I've got the services/ folder in META-INF, with a number of files including the JenaSubsystemLifecycle

-H:+TraceServiceLoaderFeature didn't help

[afs]

What's in JenaSubsystemLifcycle? It should be

Another possibility is that problem is specific to ext.xerces. It has some classpath resources under org.apache.jena.ext.xerces. Given GraalVM does class-pruning, maybe these don't get to the native image.

[jerdebsp]

In JenaSubsystemLifecycle I have the following:

org.apache.jena.riot.system.InitRIOT
org.apache.jena.sparql.system.InitARQ
org.apache.jena.rdfs.sys.InitRDFS
org.apache.jena.sys.InitJenaCore
org.apache.jena.rdfpatch.system.InitPatch
org.apache.jena.ontapi.sys.InitOntAPI
org.apache.jena.shacl.sys.InitShacl
org.apache.jena.shex.sys.InitShex
org.apache.jena.tdb1.sys.InitTDB
org.apache.jena.tdb2.sys.InitTDB2

Seems correct.

Another possibility is that problem is specific to ext.xerces

Good point. I'll check if this is a problem

[afs]

Caused by: java.lang.RuntimeException: internal error
        at org.apache.jena.ext.xerces.impl.dv.xs.XSSimpleTypeDecl.applyFacets1(XSSimpleTypeDecl.java:768)
        at org.apache.jena.ext.xerces.impl.dv.xs.BaseSchemaDVFactory.createBuiltInTypes(BaseSchemaDVFactory.java:205)
        at org.apache.jena.ext.xerces.impl.dv.xs.SchemaDVFactoryImpl.createBuiltInTypes(SchemaDVFactoryImpl.java:44)
        at org.apache.jena.ext.xerces.impl.dv.xs.SchemaDVFactoryImpl.<clinit>(SchemaDVFactoryImpl.java:39)

(unfortunately, it hides the exception stacktrace for the exception that turned into "internal error")

Chasing this and it does seem to happen because of a subclass of DatatypeException and DatatypeException loads Java resource bundles as data files from the classpath. Specifically, XMLSchemaMessages.properties in from jena-core/src/main/resources/org/apache/jena/ext/xerces/impl/msg/ (there are other properties files as well).

This doesn't prove that's the cause but it is somewhere near. A web search suggests -H:IncludeResourceBundles is needed.

---

For 5.3.0, we have been cleaning up the Xerces code incorporated in Jena, reducing it to just what Jena uses. I have just converted all ResourceBundles (#2938) that still get used to data in java source code so in the next release there will be no use of ResourceBundle anywhere.

[jerdebsp]

I tried with -H:IncludeResourceBundles and adding the bundle directly in resource-config.json. I will wait for 5.3.0 release and test again

[afs]

Jena 5.3.0 has been released.

[jerdebsp]

Seems like same issues with Jena 5.3.0. I will explore this further

[arne-bdt]

Maybe you could also try to init Jena using JenaSystem.init() from a single threaded context, to eliminate possible side effects.

[jerdebsp]

This doesn't seem to help

[afs]

What's the stacktrace when using Jena 5.3.0?

[jerdebsp]

Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.jena.ext.xerces.impl.dv.SchemaDVFactory.<clinit>(SchemaDVFactory.java:45)
        at org.apache.jena.datatypes.xsd.XSDDatatype.<init>(XSDDatatype.java:234)
        at org.apache.jena.datatypes.xsd.XSDDatatype.<init>(XSDDatatype.java:246)
        at org.apache.jena.datatypes.xsd.impl.XSDFloat.<init>(XSDFloat.java:49)
        at org.apache.jena.datatypes.xsd.XSDDatatype.<clinit>(XSDDatatype.java:54)
        at org.apache.jena.sparql.function.StandardFunctions.loadStdDefs(StandardFunctions.java:69)
        at org.apache.jena.sparql.function.FunctionRegistry.init(FunctionRegistry.java:50)
        at org.apache.jena.query.ARQ.init(ARQ.java:665)
        at org.apache.jena.sparql.system.InitARQ.start(InitARQ.java:29)
        at org.apache.jena.base.module.Subsystem.lambda$initialize$1(Subsystem.java:117)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1597)
        at org.apache.jena.base.module.Subsystem.forEach(Subsystem.java:193)
        at org.apache.jena.base.module.Subsystem.forEach(Subsystem.java:169)
        at org.apache.jena.base.module.Subsystem.initialize(Subsystem.java:115)
        at org.apache.jena.sys.JenaSystem.init(JenaSystem.java:90)
        at com.sp.odrl.shell.Application.main(Application.java:16)
        at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)
Caused by: java.lang.RuntimeException: internal error
        at org.apache.jena.ext.xerces.impl.dv.xs.XSSimpleTypeDecl.applyFacets1(XSSimpleTypeDecl.java:768)
        at org.apache.jena.ext.xerces.impl.dv.xs.BaseSchemaDVFactory.createBuiltInTypes(BaseSchemaDVFactory.java:205)
        at org.apache.jena.ext.xerces.impl.dv.xs.SchemaDVFactoryImpl.createBuiltInTypes(SchemaDVFactoryImpl.java:44)
        at org.apache.jena.ext.xerces.impl.dv.xs.SchemaDVFactoryImpl.<clinit>(SchemaDVFactoryImpl.java:39)
        ... 17 more

I am using the following pom dependency entry:

        <dependency>
            <groupId>org.apache.jena</groupId>
            <artifactId>apache-jena-libs</artifactId>
            <version>5.3.0</version>
            <type>pom</type>
        </dependency>

[afs]

XSSimpleTypeDecl was an area of change at 5.3.0 as part of the removal of unused code in the Jena copy of Xerces code.

Caused by: java.lang.RuntimeException: internal error
at org.apache.jena.ext.xerces.impl.dv.xs.XSSimpleTypeDecl.applyFacets1(XSSimpleTypeDecl.java:768)

XSSimpleTypeDecl.java:768 doesn't appear to be applyFacets1 in Jena 5.3.0 - it is now around line 277.

[Ostrzyciel]

I can confirm that Native Image compilation works with Jena 5.3.0. We use it in jelly-cli.

We only had an issue with some classes from Caffeine missing in the build (very similar to the issue described here: ben-manes/caffeine#1840). This was about the Caffeine cache used for storing blank node hashes in RDF file parsing. We resolved it by adding a manual rule instructing Graal to always include the relevant class in the build.