[SPARK-52424] Add unit test to verify in Spark Connect, Column object defined outside of an UDF can be referenced properly. #51101
+21
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Let's file a JIRA |
haoyangeng-db
changed the title
Done -- SPARK-52424. |
ueshin
reviewed
Jun 18, 2025
Comment on lines
145
to
146
| # Define 'col' outside the UDF below, so with Spark Connect it'd have to be serialized. | ||
| col = sf.lit(10) |
There was a problem hiding this comment.
This should be defined outside of func and used in func, not f.
…tside of an UDF can be serialized properly.
ueshin
reviewed
Jun 18, 2025
| @@ -136,6 +138,29 @@ def func(df, _): | |||
| if q: | |||
| q.stop() | |||
|
|
|||
| def test_streaming_foreach_batch_external_column(self): | |||
There was a problem hiding this comment.
Could you move this to pyspark/sql/tests/streaming/test_streaming_foreach_batch.py? The test will be shared between classic and connect.
There was a problem hiding this comment.
Done. Also per discussion, removed the UDF part and made 'col' defined outside of the func.
There was a problem hiding this comment.
We can remove this now as the super method will be called automatically.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Added a unit test to verify the case of a UDF referencing a Column object defined outside of it can be referenced from the UDF as intended.
Why are the changes needed?
This isn't really an intended usage pattern, but there are users implicitly relying on this behavior. So it'd cause issues for those users if this is unintentionally broken. Here we add a new unit test to ensure this case is covered.
Does this PR introduce any user-facing change?
No
How was this patch tested?
This is a unit test only change.
Was this patch authored or co-authored using generative AI tooling?
No