Uh oh!
There was an error while loading. Please reload this page.
[SPARK-37467][SQL] Consolidate whole stage and non whole stage subexpression elimination - #34727
Closed
Kimahriman wants to merge 1 commit into
Closed
[SPARK-37467][SQL] Consolidate whole stage and non whole stage subexpression elimination#34727Kimahriman wants to merge 1 commit into
Kimahriman wants to merge 1 commit into
Conversation
ContributorAuthor
@viirya. I've been playing around with this and I haven't thought of any breaking cases, but curious if there's anything you can think of or problems with this approach. Mostly trying to consolidate things before playing around with subexpression elimination inside of lambda functions. |
AmplabJenkins
commented
Nov 27, 2021
Can one of the admins verify this patch? |
Kimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
3 times, most recently
from
December 24, 2021 15:44
233de65 to
8fe5f0bCompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
January 28, 2022 20:56
8fe5f0b to
c245738CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
April 25, 2022 00:49
c245738 to
7d754a0CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
June 5, 2022 17:48
7d754a0 to
51d8657CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
September 23, 2022 23:58
51d8657 to
eec318fCompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
October 18, 2022 12:02
eec318f to
a9e5151CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
November 6, 2022 13:33
a9e5151 to
adaa8d1CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
January 1, 2023 14:40
adaa8d1 to
70b8441CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
February 4, 2023 13:37
70b8441 to
e35f5bcCompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
March 3, 2023 12:54
e35f5bc to
16d897dCompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
March 30, 2023 11:45
16d897d to
7560c20CompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
June 21, 2023 11:20
7560c20 to
0c803edCompareKimahrimanforce-pushed
the
consolidate-subexpr-elimination
branch
from
August 13, 2023 12:58
0c803ed to
e6a113aCompare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR consolidates the code paths for subexpression elimination in whole stage and non-whole stage codegen. Whole stage codegen seemed to be mostly a superset of the non-whole stage subexpression elimination, just with whole stage not using the codegen context to track subexpressions. Since subexpression values are replaced with empty blocks when evaluated, the context should be able to track the subexpressions across multiple operators. Not sure if there's corner cases I'm missing though.
It shouldn't result in any functionality changes, but there are slight differences in the generated code as a result of this:
Why are the changes needed?
Currently, there are different code paths to handle subexpression elimination in whole stage and non-whole stage codegen. This makes it harder to add new capabilities to subexpression elimination having to deal with independent code paths.
Does this PR introduce any user-facing change?
No, just slight changes in generated code.
How was this patch tested?
Existing unit tests.