Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit d1c1118

Browse files
efaracci018LuciferYang
authored andcommittedJun 6, 2025·
[SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite
### What changes were proposed in this pull request? Fix the version parsing logic in `HiveExternalCatalogVersionsSuite` to properly handle new artifact paths in https://dist.apache.org/repos/dist/release/spark/ so that "backward compatibility" test can be run. This change creates a constant `val SparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r` for more precise version matching, and removes redundant `.filterNot(_.contains("preview"))` which is no longer needed. ### Why are the changes needed? The suite is failing to execute the "backward compatibility" test due to parsing errors with testing versions. The current implementation fails to parse versions when encountering new paths like `spark-connect-swift-0.1.0/` and `spark-kubernetes-operator-0.1.0/` in https://dist.apache.org/repos/dist/release/spark/. This leads to `PROCESS_TABLES.testingVersions` being empty, and in turn a logError: "Exception encountered when invoking run on a nested suite - Fail to get the latest Spark versions to test". As a result, the condition is not met to run the test. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Executed local build and test for `HiveExternalCatalogVersionsSuite`: `build/mvn -pl sql/hive-Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite test-compile scalatest:test` Verified that the reported error no longer appears, "backward compatibility" test runs successfully, and `PROCESS_TABLES.testingVersions` now correctly contains "3.5.5" when printed out, which was previously empty. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50989 from efaracci018/fix-testingVersions. Lead-authored-by: Emilie Faracci <[email protected]> Co-authored-by: efaracci018 <[email protected]> Signed-off-by: yangjie01 <[email protected]> (cherry picked from commit a380380) Signed-off-by: yangjie01 <[email protected]>
1 parent d123aaf commit d1c1118

File tree

1 file changed

+9
-4
lines changed

1 file changed

+9
-4
lines changed
 

‎sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,12 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
9696
mirrors.distinct :+ "https://archive.apache.org/dist" :+ PROCESS_TABLES.releaseMirror
9797
logInfo(s"Trying to download Spark $version from $sites")
9898
for (site <- sites) {
99-
val filename = s"spark-$version-bin-hadoop3-scala2.13.tgz"
99+
val scalaVersion = version match {
100+
case v if v.startsWith("3.") => "-scala2.13"
101+
case v if v.startsWith("4.") => ""
102+
case _ => fail(s"Spark version $version is unexpected")
103+
}
104+
val filename = s"spark-$version-bin-hadoop3$scalaVersion.tgz"
100105
val url = s"$site/spark/spark-$version/$filename"
101106
logInfo(s"Downloading Spark $version from $url")
102107
try {
@@ -262,13 +267,13 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
262267
val testingVersions: Seq[String] = if (isPythonVersionAvailable &&
263268
SystemUtils.isJavaVersionAtMost(JavaVersion.JAVA_17)) {
264269
import scala.io.Source
270+
val sparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r
265271
try Utils.tryWithResource(
266272
Source.fromURL(s"$releaseMirror/spark")) { source =>
267273
source.mkString
268274
.split("\n")
269-
.filter(_.contains("""<a href="spark-"""))
270-
.filterNot(_.contains("preview"))
271-
.map("""<a href="spark-(\d.\d.\d)/">""".r.findFirstMatchIn(_).get.group(1))
275+
.filter(sparkVersionPattern.unanchored.matches(_))
276+
.map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
272277
.filter(_ < org.apache.spark.SPARK_VERSION)
273278
.filterNot(skipReleaseVersions.contains).toImmutableArraySeq
274279
} catch {

0 commit comments

Comments
 (0)
Please sign in to comment.