Feat kryo max buffersize by koertkuipers · Pull Request #735 · apache/spark

koertkuipers · 2014-05-11T22:08:52Z

No description provided.

…ryo Output

AmplabJenkins · 2014-05-11T22:12:58Z

Can one of the admins verify this patch?

mateiz · 2014-05-12T01:31:44Z

Hey, so is this a new feature that was recently added to Kryo? Seems super useful, but in this case, I'd actually make the max buffer size higher by default. Or we can use the old setting as a max, and create a new setting for the initial buffer size.

mateiz · 2014-05-12T01:31:51Z

Jenkins, this is ok to test

AmplabJenkins · 2014-05-12T01:32:58Z

Merged build triggered.

AmplabJenkins · 2014-05-12T01:33:04Z

Merged build started.

AmplabJenkins · 2014-05-12T02:12:23Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-12T02:12:23Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14895/

koertkuipers · 2014-05-12T22:22:11Z

hey matei,
i think they always had this feature in kryo, at least in 2.x.

created jira here:
https://issues.apache.org/jira/browse/SPARK-1811

only reason to be careful with this feature is that there is some overhead on buffer resizing (each time a new array is created with twice the size and the data is copied), which is why i was reluctant to make the old setting as a max.

how about defaults 2MB for initial and 16MB for max?

AmplabJenkins · 2014-05-14T04:12:58Z

Merged build triggered.

AmplabJenkins · 2014-05-14T04:13:04Z

Merged build started.

AmplabJenkins · 2014-05-14T04:52:43Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-14T04:52:43Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14960/

pwendell · 2014-05-28T05:46:41Z

This looks good to me! @koertkuipers mind updating the docs to include this (maybe wait until after #880 is merged tomorrow)? Also, mind prefixing the title here with SPARK-1811, this allows our merge script to detect the JIRA and link it.

mateiz · 2014-05-29T07:11:06Z

Hey one other thought, is there a reason to have the max this low? It might be good to make it even higher to deal with the odd large object (e.g. people working with large text documents). I'd go for 64 MB.

pwendell · 2014-06-04T05:48:39Z

@koertkuipers any interest in updating the docs and bumping the default? This would be a good change to have.

koertkuipers · 2014-06-20T19:18:10Z

hey sorry somehow misses this conversation thread. sure will update
defaults and docs

On Wed, Jun 4, 2014 at 1:48 AM, Patrick Wendell notifications@github.com
wrote:

@koertkuipers https://github.com/koertkuipers any interest in updating
the docs and bumping the default? This would be a good change to have.
—
Reply to this email directly or view it on GitHub
#735 (comment).

… some documentation

AmplabJenkins · 2014-06-21T17:15:04Z

Merged build triggered.

AmplabJenkins · 2014-06-21T17:15:13Z

Merged build started.

AmplabJenkins · 2014-06-21T17:52:40Z

Merged build finished.

AmplabJenkins · 2014-06-21T17:52:40Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15984/

koertkuipers · 2014-06-23T16:10:35Z

not sure if i am supposed to deal with these failures?

On Sat, Jun 21, 2014 at 1:52 PM, UCB AMPLab notifications@github.com
wrote:

Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15984/
—
Reply to this email directly or view it on GitHub
#735 (comment).

pwendell · 2014-07-07T13:14:32Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-07T13:16:03Z

Merged build triggered.

AmplabJenkins · 2014-07-07T13:16:13Z

Merged build started.

AmplabJenkins · 2014-07-07T14:00:53Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-07T14:00:53Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16372/

mingyukim · 2014-07-09T22:46:41Z

Hi all, is there any update on this PR?

koertkuipers · 2014-07-09T23:26:01Z

i updated docs and defaults as requested. currently waiting for feedback or
a merge

On Wed, Jul 9, 2014 at 6:46 PM, mingyukim notifications@github.com wrote:

Hi all, is there any update on this PR?
—
Reply to this email directly or view it on GitHub
#735 (comment).

pwendell · 2014-07-17T01:50:59Z

Jenkins, retest this please. @koertkuipers mind creating a JIRA for this (we're trying to track all changes now). I can merge this in once we have an issue for it.

SparkQA · 2014-07-17T01:52:58Z

QA tests have started for PR 735. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16760/consoleFull

koertkuipers · 2014-07-17T02:19:10Z

https://issues.apache.org/jira/browse/SPARK-2543

On Wed, Jul 16, 2014 at 9:53 PM, Apache Spark QA notifications@github.com
wrote:

QA tests have started for PR 735. This patch merges cleanly.
View progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16760/consoleFull
—
Reply to this email directly or view it on GitHub
#735 (comment).

SparkQA · 2014-07-17T03:31:02Z

QA results for PR 735:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16760/consoleFull

mateiz · 2014-07-30T18:18:20Z

Thanks for the update, Koert!

@pwendell @rxin after this, what do you guys think of lowering the initial buffer size to 1 MB or 512K? We've had some efforts to remove and reuse buffers, but not everything is doing it yet.

Author: Koert Kuipers <koert@tresata.com> Closesapache#735 from koertkuipers/feat-kryo-max-buffersize and squashes the following commits: 15f6d81 [Koert Kuipers] change default for spark.kryoserializer.buffer.max.mb to 64mb and add some documentation 1bcc22c [Koert Kuipers] Merge branch 'master' into feat-kryo-max-buffersize 0c9f8eb [Koert Kuipers] make default for kryo max buffer size 16MB 143ec4d [Koert Kuipers] test resizable buffer in kryo Output 0732445 [Koert Kuipers] support setting maxCapacity to something different than capacity in kryo Output

* KE-11534 Upgrade Janino to 3.1.6 for Calcite 1.30 * KE-11534 [Follow up]Fix for null vs true/false comparisons in TDVT SQL626 brought by Calcite 1.30

koertkuipers added 2 commits May 11, 2014 16:40

support setting maxCapacity to something different than capacity in k…
0732445
…ryo Output

test resizable buffer in kryo Output
143ec4d

make default for kryo max buffer size 16MB
0c9f8eb

koertkuipers added 2 commits June 21, 2014 12:24

Merge branch 'master' into feat-kryo-max-buffersize
1bcc22c

change default for spark.kryoserializer.buffer.max.mb to 64mb and add…
15f6d81
… some documentation

asfgit closed this in 7c5fc28Jul 30, 2014

Uh oh!

Conversation

koertkuipers commented May 11, 2014

Uh oh!

AmplabJenkins commented May 11, 2014

Uh oh!

mateiz commented May 12, 2014

Uh oh!

mateiz commented May 12, 2014

Uh oh!

AmplabJenkins commented May 12, 2014

Uh oh!

AmplabJenkins commented May 12, 2014

Uh oh!

AmplabJenkins commented May 12, 2014

Uh oh!

AmplabJenkins commented May 12, 2014

Uh oh!

koertkuipers commented May 12, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

pwendell commented May 28, 2014

Uh oh!

mateiz commented May 29, 2014

Uh oh!

pwendell commented Jun 4, 2014

Uh oh!

koertkuipers commented Jun 20, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

koertkuipers commented Jun 23, 2014

Uh oh!

pwendell commented Jul 7, 2014

Uh oh!

AmplabJenkins commented Jul 7, 2014

Uh oh!

AmplabJenkins commented Jul 7, 2014

Uh oh!

AmplabJenkins commented Jul 7, 2014

Uh oh!

AmplabJenkins commented Jul 7, 2014

Uh oh!

mingyukim commented Jul 9, 2014

Uh oh!

koertkuipers commented Jul 9, 2014

Uh oh!

pwendell commented Jul 17, 2014

Uh oh!

SparkQA commented Jul 17, 2014

Uh oh!

koertkuipers commented Jul 17, 2014

Uh oh!

SparkQA commented Jul 17, 2014

Uh oh!

mateiz commented Jul 30, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants