-
Notifications
You must be signed in to change notification settings - Fork 28.7k
[SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations #50814
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
We should make GA green first |
ece5812
to
51708f6
Compare
what if the shuffle stage is being reused by other places? I don't think we can cancel it without a ref counting to prove it's not used anywhere. |
Appreciate your time to review, could you help to explain more detail about 'reused by other places', currently i use |
I think it's more complicated than what you implemented now: what if the reuse stage is completed before we want to cancel the original stage? What if the reuse stage has not started yet? It's really a complicate algorithm with shuffle reuse being considered, can we explain it in detail in the PR description? |
Fine, update the PR description.
|
Is there a possibility for this PR to progress further? |
What changes were proposed in this pull request?
a.This pr introduce cancel queryStage mechanism.
EmptyPropagate
should always be cancelled), the queryStage has no relation with current logical plan nor physical plan any moreb.Apply this mechanism to
AQEPropagateEmptyRelation
, cancel the running queryStages which are unnecessary since propagate empty relations.Why are the changes needed?
Does this PR introduce any user-facing change?
Yes, user will see stage failure because of optimized stage cancellation , but this failure takes no effect to the query result
How was this patch tested?
Manual test, since we can not guarantee the completion order of query stages, it's not reliable to put it in unit test
before this pr


after this pr
Was this patch authored or co-authored using generative AI tooling?
No.