Skip to content

[SPARK-52482][SQL][CORE] ZStandard support for file data source reader #51182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 13 commits into from
Closed
Prev Previous commit
Next Next commit
debug
  • Loading branch information
sandip-db committed Jun 18, 2025
commit 58da8043a6d6c48cbb7872dc839c8c7981858e70
Original file line number Diff line number Diff line change
@@ -38,7 +38,8 @@ import org.apache.spark.io.{CompressionCodec => SparkCompressionCodec}
*/
object HadoopCodecStreams {
private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
private val isSparkZstdCodecEnabled = sparkConf.get(config.FILE_DATA_SOURCE_ZSTANDARD_ENABLED)
private val isSparkZstdCodecEnabled =
sparkConf.getBoolean(config.FILE_DATA_SOURCE_ZSTANDARD_ENABLED.key, true)
private val ZSTD_EXTENSIONS = Seq(".zstd", ".zst")

// get codec based on file name extension