Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit 73a3b1c

Browse files
committedMay 24, 2025·
[SPARK-52290][SQL][BUILD] Fix the Maven compilation and testing issues for sql/pipelines module
### What changes were proposed in this pull request? The following modifications are mainly made in this PR: 1. Add the dependency `org.scala-lang:scala-library` to fix: ``` [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.2:doc-jar (attach-scaladocs) on project spark-pipelines_2.13: MavenReportException: Error while creating archive: wrap: Process exited with an error: 1 (Exit value: 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :spark-pipelines_2.13 ``` 2. Add the dependency `org.apache.spark:spark-core` to fix: ``` *** RUN ABORTED *** A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/spark/internal/config/Tests$ java.lang.NoClassDefFoundError: org/apache/spark/internal/config/Tests$ at org.apache.spark.SparkFunSuite.beforeAll(SparkFunSuite.scala:94) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178) at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225) at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1324) at org.scalatest.Suite.runNestedSuites(Suite.scala:1223) at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156) ... Cause: java.lang.ClassNotFoundException: org.apache.spark.internal.config.Tests$ at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at org.apache.spark.SparkFunSuite.beforeAll(SparkFunSuite.scala:94) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178) at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225) ``` ### Why are the changes needed? Fix the Maven compilation and testing issues for `sql/pipelines` module, maven daily test failed after SPARK-52166 - https://github.com/apache/spark/actions/runs/15212144791 - https://github.com/apache/spark/actions/runs/15211074246 - https://github.com/apache/spark/actions/runs/15213443142 - https://github.com/apache/spark/actions/runs/15218341216 ![image](https://github.com/user-attachments/assets/70379d0f-cf13-4337-a538-791dd21667bb) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Local check: ``` build/mvn clean install -pl sql/pipelines -am -DskipTests build/mvn test -pl sql/pipelines ``` ``` Run completed in 356 milliseconds. Total number of tests run: 9 Suites: completed 3, aborted 0 Tests: succeeded 9, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #51008 from LuciferYang/SPARK-52290. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
1 parent cf3ef31 commit 73a3b1c

File tree

1 file changed

+10
-0
lines changed

1 file changed

+10
-0
lines changed
 

‎sql/pipelines/pom.xml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,16 @@
3232
<name>Spark Project Declarative Pipelines Library</name>
3333
<url>https://spark.apache.org/</url>
3434
<dependencies>
35+
<dependency>
36+
<groupId>org.scala-lang</groupId>
37+
<artifactId>scala-library</artifactId>
38+
</dependency>
39+
<dependency>
40+
<groupId>org.apache.spark</groupId>
41+
<artifactId>spark-core_${scala.binary.version}</artifactId>
42+
<version>${project.version}</version>
43+
<scope>test</scope>
44+
</dependency>
3545
<dependency>
3646
<groupId>org.apache.spark</groupId>
3747
<artifactId>spark-core_${scala.binary.version}</artifactId>

0 commit comments

Comments
 (0)
Please sign in to comment.