[SPARK-3577] Report Spill size on disk for UnsafeExternalSorter by sitalkedia · Pull Request #17471 · apache/spark

sitalkedia · 2017-03-29T18:46:41Z

What changes were proposed in this pull request?

Report Spill size on disk for UnsafeExternalSorter

How was this patch tested?

Tested by running a job on cluster and verify the spill size on disk.

SparkQA · 2017-03-29T21:27:12Z

Test build #75363 has finished for PR 17471 at commit 4546d43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sitalkedia · 2017-04-05T01:19:09Z

cc - @rxin, @kayousterhout, @squito

rxin · 2017-04-05T04:15:51Z

cc @cloud-fan / @ueshin / @sameeragarwal can you review this?

cloud-fan · 2017-04-05T08:46:00Z

SPARK-3577 is about reporting spill time, seems not related to this PR?

sitalkedia · 2017-04-05T15:07:44Z

@cloud-fan - There is discussion in the JIRA to report the disk spill size as well. I can create a new JIRA if you prefer.

squito · 2017-04-05T15:18:44Z

this is basically the same as this old pr, right? #15616

the last comment on there is a request for a test, which I think still applies here. I also think a separate jira would be clearer.

sitalkedia · 2017-04-05T17:45:22Z

I did not see we already have an open PR for this. Sure, I will add test to this PR and also file a separate JIRA.

sameeragarwal · 2017-05-25T07:25:58Z

@sitalkedia any updates here?

sitalkedia · 2017-05-25T18:06:21Z

@sameeragarwal - Thanks for taking a look. I will update the PR adding test case soon.

HyukjinKwon · 2017-06-19T05:11:08Z

I was just looking though PRs for my curiosity. Please let me leave a gentle ping @sitalkedia.

SparkQA · 2017-06-21T01:12:40Z

Test build #78348 has started for PR 17471 at commit da1f384.

sitalkedia · 2017-06-21T01:13:18Z

Thanks for the ping @HyukjinKwon. Updated the PR with test case.
cc - @sameeragarwal

shaneknapp · 2017-06-21T04:21:04Z

test this please

SparkQA · 2017-06-21T04:23:31Z

Test build #78363 has started for PR 17471 at commit da1f384.

sitalkedia · 2017-06-21T19:55:19Z

Jenkins retest this please.

sitalkedia · 2017-06-21T21:49:56Z

@shaneknapp, @sameeragarwal - Not able to access the jenkins build link- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78405/testReport, something wrong with the jenkins sever?

kayousterhout · 2017-06-21T21:52:07Z

I think the power is still out in the CS building at Berkeley because of the earthquake, so I'm guessing Jenkins is down as a result (note that even the vanilla AMP website doesn't work: http://amplab.cs.berkeley.edu/)

shaneknapp · 2017-06-21T22:24:02Z

it was actually a cut power line due to street construction. apparently power is back online, and we're waiting for machines and switches to finish booting back up. i'm WFH today, so i'm monitoring things remotely.

…

On Wed, Jun 21, 2017 at 2:52 PM, Kay Ousterhout ***@***.***> wrote: I think the power is still out in the CS building at Berkeley because of the earthquake, so I'm guessing Jenkins is down as a result (note that even the vanilla AMP website doesn't work: http://amplab.cs.berkeley.edu/) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17471 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABiDrCpWyketc1oi9h0i7qKbvgEEVIbnks5sGZCtgaJpZM4MtZcI> .

shaneknapp · 2017-06-21T22:24:37Z

also, see my message to the dev@ list. jenkins is up, but not reachable via the amplab link.

…

On Wed, Jun 21, 2017 at 3:23 PM, shane knapp ☠ ***@***.***> wrote: it was actually a cut power line due to street construction. apparently power is back online, and we're waiting for machines and switches to finish booting back up. i'm WFH today, so i'm monitoring things remotely. On Wed, Jun 21, 2017 at 2:52 PM, Kay Ousterhout ***@***.***> wrote: > I think the power is still out in the CS building at Berkeley because of > the earthquake, so I'm guessing Jenkins is down as a result (note that even > the vanilla AMP website doesn't work: http://amplab.cs.berkeley.edu/) > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#17471 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ABiDrCpWyketc1oi9h0i7qKbvgEEVIbnks5sGZCtgaJpZM4MtZcI> > . >

SparkQA · 2017-06-21T22:46:50Z

Test build #78405 has finished for PR 17471 at commit da1f384.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-06-26T12:24:43Z

+ for (int i = 0; i < n; i++) {
+ record[0] = (long) i;
+ sorter.insertRecord(record, Platform.LONG_ARRAY_OFFSET, recordSize, 0, false);
+ }


can you add some comments to explain the math here? I don't quite understand why it should be >= 2...

Added a comment to make it clear.

cloud-fan · 2017-06-26T12:37:05Z

LGTM, shall we fix it for other sorters? e.g. UnsafeExternalRowSorter, UnsafeKVExternalSorter, etc.

jiangxb1987 · 2017-06-26T15:08:48Z

 inMemSorter.free();
 inMemSorter = null;
 taskContext.taskMetrics().incMemoryBytesSpilled(released);
+ taskContext.taskMetrics().incDiskBytesSpilled(writeMetrics.bytesWritten());


Why do we create a new writeMetrics here, instead of report this.writeMetrics.bytesWritten?

We're spilling, so bytes written should be counted towards spill rather than write.
Approach similar to - https://github.com/sitalkedia/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java#L138

sitalkedia · 2017-06-28T15:04:56Z

shall we fix it for other sorters? e.g. UnsafeExternalRowSorter, UnsafeKVExternalSorter, etc.

Actually other sorters are using UnsafeExternalSorter internally(https://github.com/sitalkedia/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java#L67), so this change should fix others as well.

SparkQA · 2017-06-28T21:46:42Z

Test build #78821 has finished for PR 17471 at commit 6b94c2b.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

sitalkedia · 2017-06-28T22:25:24Z

Jenkins retest this please.

SparkQA · 2017-06-29T00:54:26Z

Test build #78830 has finished for PR 17471 at commit 6b94c2b.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-06-29T03:05:50Z

retest this please

SparkQA · 2017-06-29T05:53:08Z

Test build #78851 has finished for PR 17471 at commit 6b94c2b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-06-29T06:26:13Z

thanks, merging to master!

## What changes were proposed in this pull request? Report Spill size on disk for UnsafeExternalSorter ## How was this patch tested? Tested by running a job on cluster and verify the spill size on disk. Author: Sital Kedia <skedia@fb.com> Closesapache#17471 from sitalkedia/fix_disk_spill_size.

Sital Kedia added 2 commits June 20, 2017 17:55

[SPARK-3577] Report Spill size on disk for UnsafeExternalSorter
33143de

Add test
da1f384

sitalkedia force-pushed the fix_disk_spill_size branch from 4546d43 to da1f384Compare June 21, 2017 01:11

cloud-fan reviewed Jun 26, 2017
View reviewed changes

jiangxb1987 reviewed Jun 26, 2017
View reviewed changes

Minor test change
6b94c2b

asfgit closed this in a946be3Jun 29, 2017

Uh oh!

Conversation

sitalkedia commented Mar 29, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 29, 2017

Uh oh!

sitalkedia commented Apr 5, 2017

Uh oh!

rxin commented Apr 5, 2017

Uh oh!

cloud-fan commented Apr 5, 2017

Uh oh!

sitalkedia commented Apr 5, 2017

Uh oh!

squito commented Apr 5, 2017

Uh oh!

sitalkedia commented Apr 5, 2017

Uh oh!

sameeragarwal commented May 25, 2017

Uh oh!

sitalkedia commented May 25, 2017

Uh oh!

HyukjinKwon commented Jun 19, 2017

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

sitalkedia commented Jun 21, 2017

Uh oh!

shaneknapp commented Jun 21, 2017

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

sitalkedia commented Jun 21, 2017

Uh oh!

sitalkedia commented Jun 21, 2017

Uh oh!

kayousterhout commented Jun 21, 2017

Uh oh!

shaneknapp commented Jun 21, 2017 via email

Uh oh!

shaneknapp commented Jun 21, 2017 via email

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

cloud-fanJun 26, 2017

Choose a reason for hiding this comment

Uh oh!

sitalkediaJun 28, 2017

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jun 26, 2017

Uh oh!

jiangxb1987Jun 26, 2017

Choose a reason for hiding this comment

Uh oh!

sitalkediaJun 28, 2017

Choose a reason for hiding this comment

Uh oh!

sitalkedia commented Jun 28, 2017

Uh oh!

SparkQA commented Jun 28, 2017

Uh oh!

sitalkedia commented Jun 28, 2017

Uh oh!

SparkQA commented Jun 29, 2017

Uh oh!

cloud-fan commented Jun 29, 2017

Uh oh!

SparkQA commented Jun 29, 2017

Uh oh!

cloud-fan commented Jun 29, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development