Uh oh!
There was an error while loading. Please reload this page.
[SPARK-21258][SQL] Fix WindowExec complex object aggregation with spilling - #18470
Closed
hvanhovell wants to merge 2 commits into
Closed
[SPARK-21258][SQL] Fix WindowExec complex object aggregation with spilling#18470hvanhovell wants to merge 2 commits into
hvanhovell wants to merge 2 commits into
Conversation
gatorsmile
commented
Jun 29, 2017
Member
LGTM @liancheng@cloud-fan |
liancheng
commented
Jun 29, 2017
Contributor
LGTM pending Jenkins. |
SparkQA
commented
Jun 29, 2017
Test build #78932 has finished for PR 18470 at commit
|
SparkQA
commented
Jun 29, 2017
Test build #78933 has finished for PR 18470 at commit
|
cloud-fan
commented
Jun 30, 2017
Contributor
retes this please |
cloud-fan
commented
Jun 30, 2017
Contributor
retest this please |
cloud-fan
commented
Jun 30, 2017
Contributor
LGTM |
SparkQA
commented
Jun 30, 2017
Test build #78941 has finished for PR 18470 at commit
|
asfgit pushed a commit
that referenced
this pull request
Jun 30, 2017
…lling ## What changes were proposed in this pull request? `WindowExec` currently improperly stores complex objects (UnsafeRow, UnsafeArrayData, UnsafeMapData, UTF8String) during aggregation by keeping a reference in the buffer used by `GeneratedMutableProjections` to the actual input data. Things go wrong when the input object (or the backing bytes) are reused for other things. This could happen in window functions when it starts spilling to disk. When reading the back the spill files the `UnsafeSorterSpillReader` reuses the buffer to which the `UnsafeRow` points, leading to weird corruption scenario's. Note that this only happens for aggregate functions that preserve (parts of) their input, for example `FIRST`, `LAST`, `MIN` & `MAX`. This was not seen before, because the spilling logic was not doing actual spills as much and actually used an in-memory page. This page was not cleaned up during window processing and made sure unsafe objects point to their own dedicated memory location. This was changed by #16909, after this PR Spark spills more eagerly. This PR provides a surgical fix because we are close to releasing Spark 2.2. This change just makes sure that there cannot be any object reuse at the expensive of a little bit of performance. We will follow-up with a more subtle solution at a later point. ## How was this patch tested? Added a regression test to `DataFrameWindowFunctionsSuite`. Author: Herman van Hovell <hvanhovell@databricks.com> Closes#18470 from hvanhovell/SPARK-21258. (cherry picked from commit e2f32ee) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan
commented
Jun 30, 2017
Contributor
thanks, merging to master/2.2/2.1! |
asfgit pushed a commit
that referenced
this pull request
Jun 30, 2017
…lling ## What changes were proposed in this pull request? `WindowExec` currently improperly stores complex objects (UnsafeRow, UnsafeArrayData, UnsafeMapData, UTF8String) during aggregation by keeping a reference in the buffer used by `GeneratedMutableProjections` to the actual input data. Things go wrong when the input object (or the backing bytes) are reused for other things. This could happen in window functions when it starts spilling to disk. When reading the back the spill files the `UnsafeSorterSpillReader` reuses the buffer to which the `UnsafeRow` points, leading to weird corruption scenario's. Note that this only happens for aggregate functions that preserve (parts of) their input, for example `FIRST`, `LAST`, `MIN` & `MAX`. This was not seen before, because the spilling logic was not doing actual spills as much and actually used an in-memory page. This page was not cleaned up during window processing and made sure unsafe objects point to their own dedicated memory location. This was changed by #16909, after this PR Spark spills more eagerly. This PR provides a surgical fix because we are close to releasing Spark 2.2. This change just makes sure that there cannot be any object reuse at the expensive of a little bit of performance. We will follow-up with a more subtle solution at a later point. ## How was this patch tested? Added a regression test to `DataFrameWindowFunctionsSuite`. Author: Herman van Hovell <hvanhovell@databricks.com> Closes#18470 from hvanhovell/SPARK-21258. (cherry picked from commit e2f32ee) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Contributor
Hi, @cloud-fan , @hvanhovell , after merging this pr into branch-2.1, there are some errors: |
cloud-fan
commented
Jun 30, 2017
Contributor
ah the spilling logic is not in 2.1, let me revert it, sorry for the trouble. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
WindowExeccurrently improperly stores complex objects (UnsafeRow, UnsafeArrayData, UnsafeMapData, UTF8String) during aggregation by keeping a reference in the buffer used byGeneratedMutableProjectionsto the actual input data. Things go wrong when the input object (or the backing bytes) are reused for other things. This could happen in window functions when it starts spilling to disk. When reading the back the spill files theUnsafeSorterSpillReaderreuses the buffer to which theUnsafeRowpoints, leading to weird corruption scenario's. Note that this only happens for aggregate functions that preserve (parts of) their input, for exampleFIRST,LAST,MIN&MAX.This was not seen before, because the spilling logic was not doing actual spills as much and actually used an in-memory page. This page was not cleaned up during window processing and made sure unsafe objects point to their own dedicated memory location. This was changed by #16909, after this PR Spark spills more eagerly.
This PR provides a surgical fix because we are close to releasing Spark 2.2. This change just makes sure that there cannot be any object reuse at the expensive of a little bit of performance. We will follow-up with a more subtle solution at a later point.
How was this patch tested?
Added a regression test to
DataFrameWindowFunctionsSuite.