[SPARK-6979][Streaming] Replace JobScheduler.eventActor and JobGenerator.eventActor with EventLoop by zsxwing · Pull Request #5554 · apache/spark

zsxwing · 2015-04-17T10:16:13Z

Title says it all.

srowen · 2015-04-17T11:09:09Z

If you have a moment, this is more for my edification, is the benefit simply that there is no need to participate in the full-blown actor system, and that this is simpler to reason about as a local event loop? are there other benefits or drawabacks?

SparkQA · 2015-04-17T11:28:42Z

Test build #30480 has finished for PR 5554 at commit ec6ec58.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

zsxwing · 2015-04-17T12:04:26Z

@srowen this is a part of SPARK-5293. The future plan is make Spark be able to be built without Akka at all eventually.

zsxwing · 2015-04-17T12:05:05Z

retest this please

SparkQA · 2015-04-17T12:20:38Z

Test build #30479 has finished for PR 5554 at commit ce0fa73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-17T13:49:46Z

Test build #30482 has finished for PR 5554 at commit ec6ec58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-17T14:44:24Z

Test build #30485 has finished for PR 5554 at commit e496ace.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

zsxwing · 2015-04-17T16:18:40Z

I just replaced jobScheduler.reportError("Error in job generator", e) with logError("Error in job generator", e) and NotSerializableException disappeared. Not sure why. Will dig deeply tomorrow.

rxin · 2015-04-17T16:26:02Z

add a blank line here

SparkQA · 2015-04-17T17:42:53Z

Test build #30486 has finished for PR 5554 at commit 633e279.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

rxin · 2015-04-19T00:30:51Z

Thanks - please ping me when this is ready.

zsxwing · 2015-04-19T17:38:45Z

Finally, figured out why logError("Error in job generator", e) won't fail the test cases.

NotSerializableException also happens in the old codes, but it just forces the Actor restart because it's not caught by any code. Then I use EventLoop to replace Actor, so NotSerializableException will be caught and sent to the onError method. If I use logError("Error in job generator", e), NotSerializableException is just logged and doesn't fail the tests. But if I use jobScheduler.reportError("Error in job generator", e), the error will be propagated to the mail thread of the tests. That's why the tests fail.

StreamingKMeans must be Serializable since it will be used in the closure.

So it's a bug that was hidden by Actor error mechanism. And now jobScheduler.reportError exposes the bug.

zsxwing · 2015-04-19T17:40:53Z

Just added Serializable to StreamingKMeans. @rxin if you think I need to open a JIRA for the NotSerializableException bug in StreamingKMeans, please let me know.

zsxwing · 2015-04-19T17:59:48Z

I think I should create a new PR for the NotSerializableException bug in StreamingKMeans so that we can merge it to branch 1.2 and 1.3. Added #5582 for the bug.

zsxwing · 2015-04-19T18:00:48Z

Will update this PR once #5582 is merged.

SparkQA · 2015-04-19T19:28:21Z

Test build #30565 has finished for PR 5554 at commit 5304350.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

rxin · 2015-04-20T03:48:48Z

LGTM. I'm going to merge it in master since it doesn't conflict.

…tor.eventActor with EventLoop Title says it all. cc rxin tdas Author: zsxwing <zsxwing@gmail.com> Closesapache#5554 from zsxwing/SPARK-6979 and squashes the following commits: 5304350 [zsxwing] Fix NotSerializableException e9d3479 [zsxwing] Add blank lines 633e279 [zsxwing] Fix NotSerializableException e496ace [zsxwing] Replace JobGenerator.eventActor with EventLoop ec6ec58 [zsxwing] Fix the import order ce0fa73 [zsxwing] Replace JobScheduler.eventActor with EventLoop

zsxwing added 2 commits April 17, 2015 18:08

Replace JobScheduler.eventActor with EventLoop
ce0fa73

Fix the import order
ec6ec58

Replace JobGenerator.eventActor with EventLoop
e496ace

zsxwing changed the title ~~[SPARK-6979][Streaming] Replace JobScheduler.eventActor with EventLoop~~[SPARK-6979][Streaming] Replace JobScheduler.eventActor and JobGenerator.eventActor with EventLoopApr 17, 2015

Fix NotSerializableException
633e279

rxin reviewed Apr 17, 2015
View reviewed changes

Comment threadstreaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala Outdated
rxinApr 17, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add a blank line here

Add blank lines
e9d3479

Fix NotSerializableException
5304350

asfgit closed this in c776ee8Apr 20, 2015

zsxwing deleted the SPARK-6979 branch April 20, 2015 09:43

Uh oh!

Conversation

zsxwing commented Apr 17, 2015

Uh oh!

srowen commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

zsxwing commented Apr 17, 2015

Uh oh!

zsxwing commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

zsxwing commented Apr 17, 2015

Uh oh!

rxinApr 17, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

rxin commented Apr 19, 2015

Uh oh!

zsxwing commented Apr 19, 2015

Uh oh!

zsxwing commented Apr 19, 2015

Uh oh!

zsxwing commented Apr 19, 2015

Uh oh!

zsxwing commented Apr 19, 2015

Uh oh!

SparkQA commented Apr 19, 2015

Uh oh!

rxin commented Apr 20, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants