Skip to content

Commit cb2732d

Browse files
committed
[SPARK-51146][INFRA][FOLLOWUP] Use awk to update release scripts
### What changes were proposed in this pull request? This PR has two changes: 1. use awk to update the scripts instead of sed, to avoid issues on Mac OS 2. insert the `export SPARK_CONNECT_MODE=1` after the first line, instead of before the last line. `spark-shell` does not put the final command invocation at the last line. ### Why are the changes needed? fix release scripts ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? manual ### Was this patch authored or co-authored using generative AI tooling? no Closes #49891 from cloud-fan/release. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent 42ecabf commit cb2732d

File tree

1 file changed

+8
-3
lines changed

1 file changed

+8
-3
lines changed

dev/make-distribution.sh

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -317,9 +317,14 @@ if [ "$MAKE_TGZ" == "true" ]; then
317317
TARDIR="$SPARK_HOME/$TARDIR_NAME"
318318
rm -rf "$TARDIR"
319319
cp -r "$DISTDIR" "$TARDIR"
320-
sed -i -e '$s/.*/export SPARK_CONNECT_MODE=1\n&/' "$TARDIR/bin/pyspark"
321-
sed -i -e '$s/.*/export SPARK_CONNECT_MODE=1\n&/' "$TARDIR/bin/spark-shell"
322-
sed -i -e '$s/.*/export SPARK_CONNECT_MODE=1\n&/' "$TARDIR/bin/spark-submit"
320+
# Set the Spark Connect system variable in these scripts to enable it by default.
321+
awk 'NR==1{print; print "export SPARK_CONNECT_MODE=1"; next} {print}' "$TARDIR/bin/pyspark" > tmp && cat tmp > "$TARDIR/bin/pyspark"
322+
awk 'NR==1{print; print "export SPARK_CONNECT_MODE=1"; next} {print}' "$TARDIR/bin/spark-shell" > tmp && cat tmp > "$TARDIR/bin/spark-shell"
323+
awk 'NR==1{print; print "export SPARK_CONNECT_MODE=1"; next} {print}' "$TARDIR/bin/spark-submit" > tmp && cat tmp > "$TARDIR/bin/spark-submit"
324+
awk 'NR==1{print; print "set SPARK_CONNECT_MODE=1"; next} {print}' "$TARDIR/bin/pyspark2.cmd" > tmp && cat tmp > "$TARDIR/bin/pyspark2.cmd"
325+
awk 'NR==1{print; print "set SPARK_CONNECT_MODE=1"; next} {print}' "$TARDIR/bin/spark-shell2.cmd" > tmp && cat tmp > "$TARDIR/bin/spark-shell2.cmd"
326+
awk 'NR==1{print; print "set SPARK_CONNECT_MODE=1"; next} {print}' "$TARDIR/bin/spark-submit2.cmd" > tmp && cat tmp > "$TARDIR/bin/spark-submit2.cmd"
327+
rm tmp
323328
$TAR -czf "$TARDIR_NAME.tgz" -C "$SPARK_HOME" "$TARDIR_NAME"
324329
rm -rf "$TARDIR"
325330
fi

0 commit comments

Comments
 (0)