[SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts. by pwendell · Pull Request #1172 · apache/spark

pwendell · 2014-06-21T23:06:23Z

SPARK-1112: This is a more conservative version of #1132 that doesn't change
around the actor system initialization on the executor. Instead we just
directly read the current frame size limit from the ActorSystem.

SPARK-2156: This uses the same fixe as in #1132.

…head amounts. SPARK-1112: This is a more conservative version of apache#1132 that doesn't change around the actor system initialization on the executor. Instead we just directly read the current frame size limit from the ActorSystem. SPARK-2156: This uses the same fixe as in apache#1132.

pwendell · 2014-06-21T23:06:59Z

@mengxr - do you mind reviewing this?

AmplabJenkins · 2014-06-21T23:10:05Z

Merged build triggered.

AmplabJenkins · 2014-06-21T23:10:10Z

Merged build started.

AmplabJenkins · 2014-06-21T23:11:09Z

Merged build finished.

AmplabJenkins · 2014-06-21T23:11:09Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15998/

mengxr · 2014-06-22T01:43:01Z

Jenkins, retest it please.

mengxr · 2014-06-22T01:45:06Z

LGTM. I will merge it to branch-1.0 if Jenkins is happy.

aarondav · 2014-06-22T01:55:06Z

Slightly confused on this -- won't this make LocalBackend and MesosExecutorBackend never use the BlockManager for results, causing them to have the undesired behavior?

The MesosExecutorBackend sends results through mesos, not akka. The LocalBackend sends a message to an actor within the same actor system... which I assumed won't go over TCP.

I see. So the only real change is that in certain cases the LocalBackend will no longer use the BlockManager for returning results. Sounds fine.

So that change actually alters the expectations of the unit tests, so I went ahead and just enforced the limit in the LocalBackend anwyays.

pwendell · 2014-06-22T06:21:40Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-22T06:25:06Z

Merged build triggered.

AmplabJenkins · 2014-06-22T06:25:13Z

Merged build started.

AmplabJenkins · 2014-06-22T07:00:14Z

Merged build finished.

AmplabJenkins · 2014-06-22T07:00:14Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16007/

mengxr · 2014-06-22T08:12:51Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-22T08:15:06Z

Merged build triggered.

AmplabJenkins · 2014-06-22T08:15:13Z

Merged build started.

AmplabJenkins · 2014-06-22T08:51:26Z

Merged build finished.

AmplabJenkins · 2014-06-22T08:51:27Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16010/

pwendell · 2014-06-22T19:02:37Z

I think these are failing because our tests assume that in local mode we enforce the frame size limit (which we actually don't need to). I'll make the appropriate adjustments in a bit.

pwendell · 2014-06-22T21:35:57Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-22T21:40:08Z

Merged build triggered.

AmplabJenkins · 2014-06-22T21:40:18Z

Merged build started.

AmplabJenkins · 2014-06-22T22:19:13Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-22T22:19:13Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16014/

pwendell · 2014-06-23T02:26:57Z

@aarondav mind taking a final pass and merging this?

aarondav · 2014-06-23T02:30:27Z

Absolutely. LGTM, merging into branch-1.0.

…head amounts. SPARK-1112: This is a more conservative version of #1132 that doesn't change around the actor system initialization on the executor. Instead we just directly read the current frame size limit from the ActorSystem. SPARK-2156: This uses the same fixe as in #1132. Author: Patrick Wendell <pwendell@gmail.com> Closes#1172 from pwendell/akka-10-fix and squashes the following commits: d56297e [Patrick Wendell] Set limit in LocalBackend to preserve test expectations 9f5ed19 [Patrick Wendell] [SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts.

aarondav · 2014-06-23T02:42:38Z

You may have to close this manually, @pwendell, I'm not sure github will close it if it's not in master.

pwendell · 2014-06-23T02:48:55Z

Thanks, closed.

…head amounts. backport #1172 to branch-0.9. Author: Patrick Wendell <pwendell@gmail.com> Closes#1455 from mengxr/akka-fix-0.9 and squashes the following commits: a99f201 [Patrick Wendell] backport PR #1172 to branch-0.9

aarondav reviewed Jun 22, 2014
View reviewed changes

Set limit in LocalBackend to preserve test expectations
d56297e

pwendell closed this Jun 23, 2014

mengxr pushed a commit to mengxr/spark that referenced this pull request Jul 17, 2014
backport PRapache#1172to branch-0.9
a99f201

mengxr mentioned this pull request Jul 17, 2014
[SPARK-1112, 2156] (0.9 edition) Use correct akka frame size and overhead amounts. #1455
Closed

Uh oh!

Conversation

pwendell commented Jun 21, 2014

Uh oh!

pwendell commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

AmplabJenkins commented Jun 21, 2014

Uh oh!

mengxr commented Jun 22, 2014

Uh oh!

mengxr commented Jun 22, 2014

Uh oh!

aarondavJun 22, 2014

Choose a reason for hiding this comment

Uh oh!

pwendellJun 22, 2014

Choose a reason for hiding this comment

Uh oh!

aarondavJun 22, 2014

Choose a reason for hiding this comment

Uh oh!

pwendellJun 22, 2014

Choose a reason for hiding this comment

Uh oh!

pwendell commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

mengxr commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

pwendell commented Jun 22, 2014

Uh oh!

pwendell commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

AmplabJenkins commented Jun 22, 2014

Uh oh!

pwendell commented Jun 23, 2014

Uh oh!

aarondav commented Jun 23, 2014

Uh oh!

aarondav commented Jun 23, 2014

Uh oh!

pwendell commented Jun 23, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants