Uh oh!
There was an error while loading. Please reload this page.
[GLUTEN-3559][CORE] Fix output partitioning of Hash join - #4342
Conversation
Run Gluten Clickhouse CI |
ayushi-agarwal
commented
Jan 9, 2024
Run Gluten Clickhouse CI |
ayushi-agarwal
commented
Jan 10, 2024
@JkSelf@zhli1142015 The build is green. Kindly review. |
There was a problem hiding this comment.
HashJoinLikeExecTransformer is also extended by ShuffledHashJoinExecTransformer. Do we need this fix for SHJ?
There was a problem hiding this comment.
Yes, I was looking into that, this change will help it there also so I think we can keep it for SHJ also.
ulysses-you
commented
Jan 11, 2024
it seems the code is difference since Spark3.4, shall we move this to shim module ? |
There was a problem hiding this comment.
I have added a ToDO here, for this https://issues.apache.org/jira/browse/SPARK-45882 we would need to add in shim. Shall that be taken in separate PR?
This PR make the changes which were added in spark 3.1 apache/spark#28676
There was a problem hiding this comment.
@ulysses-you Shall we move this to shim module as part of this in separate PR?
There was a problem hiding this comment.
I'd like to solve this todo in this pr, as we need to move the whole code to shim module.
84f5dfe to
90aec14CompareRun Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
ulysses-you
left a comment
There was a problem hiding this comment.
thank you @ayushi-agarwal , lgtm. cc @rui-mo if you have other comments
zzcclp
commented
Jan 12, 2024
LGTM |
Thank you @zzcclp@ulysses-you@zhli1142015 for reviewing. |
What changes were proposed in this pull request?
As part of this ticket https://issues.apache.org/jira/browse/SPARK-31869 an improvement was added which can decrease the number of exchanges. As output partitioning is overridden in gluten, the test was failing as extra exchange was coming.
(Fixes: #3559)
How was this patch tested?
Ran UT locally.