[SPARK-1332] Improve Spark Streaming's Network Receiver and InputDStream API [WIP] by tdas · Pull Request #300 · apache/spark

tdas · 2014-04-02T06:24:04Z

The current Network Receiver API makes it slightly complicated to right a new receiver as one needs to create an instance of BlockGenerator as shown in SocketReceiver
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala#L51

Exposing the BlockGenerator interface has made it harder to improve the receiving process. The API of NetworkReceiver (which was not a very stable API anyways) needs to be change if we are to ensure future stability.

Additionally, the functions like streamingContext.socketStream that create input streams, return DStream objects. That makes it hard to expose functionality (say, rate limits) unique to input dstreams. They should return InputDStream or NetworkInputDStream. This is still not yet implemented.

This PR is blocked on the graceful shutdown PR #247

…s and traits to make them more testable.

AmplabJenkins · 2014-04-02T06:27:23Z

Merged build triggered.

AmplabJenkins · 2014-04-02T06:29:01Z

Merged build started.

AmplabJenkins · 2014-04-02T07:23:56Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-02T07:23:56Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13671/

…iver-api Conflicts: external/mqtt/src/main/scala/org/apache/spark/streaming/mqtt/MQTTInputDStream.scala streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/dstream/NetworkInputDStream.scala streaming/src/main/scala/org/apache/spark/streaming/scheduler/NetworkInputTracker.scala streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala

…orkInputDStream, for both Scala and Java.

tdas · 2014-04-16T19:24:47Z

@pwendell Ready for review!

AmplabJenkins · 2014-04-16T19:28:12Z

Merged build triggered.

AmplabJenkins · 2014-04-16T19:28:20Z

Merged build started.

AmplabJenkins · 2014-04-16T19:31:03Z

Merged build finished.

AmplabJenkins · 2014-04-16T19:31:03Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14184/

AmplabJenkins · 2014-04-16T19:43:11Z

Merged build triggered.

AmplabJenkins · 2014-04-16T19:43:20Z

Merged build started.

AmplabJenkins · 2014-04-16T19:46:47Z

Merged build finished.

AmplabJenkins · 2014-04-16T19:46:48Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14185/

tdas · 2014-04-16T20:08:06Z

Jenkins, test this again.

tdas · 2014-04-16T22:31:58Z

Jenkins, test this.

rxin · 2014-04-17T00:37:02Z

Jenkins, retest this please.

AmplabJenkins · 2014-04-17T00:38:14Z

Merged build triggered.

AmplabJenkins · 2014-04-17T00:38:22Z

Merged build started.

AmplabJenkins · 2014-04-17T02:04:14Z

Merged build finished.

AmplabJenkins · 2014-04-17T02:04:14Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14193/

tdas · 2014-04-17T02:07:20Z

Let me look at the failed unit tests. Should not affect PR too much, I think.

AmplabJenkins · 2014-04-19T21:23:11Z

Build triggered.

AmplabJenkins · 2014-04-19T21:23:19Z

Build started.

AmplabJenkins · 2014-04-19T21:24:27Z

Build finished.

AmplabJenkins · 2014-04-19T21:24:27Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14260/

tdas · 2014-04-21T17:37:46Z

Jenkins, test this again.

tdas · 2014-04-21T18:01:16Z

Jenkins test this

pwendell · 2014-04-21T18:04:56Z

Jenkins, test this please.

pwendell · 2014-04-21T18:05:17Z

@tdas the tests are failing becuase this doesn't merge properly.

AmplabJenkins · 2014-04-21T18:07:57Z

Build triggered.

AmplabJenkins · 2014-04-21T18:08:02Z

Build started.

AmplabJenkins · 2014-04-21T18:08:50Z

Build finished.

AmplabJenkins · 2014-04-21T18:08:50Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14289/

…iver-api

AmplabJenkins · 2014-04-21T18:22:55Z

Merged build triggered.

AmplabJenkins · 2014-04-21T18:23:03Z

Merged build started.

AmplabJenkins · 2014-04-21T19:01:31Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-21T19:01:32Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14291/

tdas · 2014-04-21T19:30:40Z

@pwendell Can you take another look at the patch and merge it in. :)

pwendell · 2014-04-22T02:05:19Z

Thanks - I've merged this!

…eam API [WIP] The current Network Receiver API makes it slightly complicated to right a new receiver as one needs to create an instance of BlockGenerator as shown in SocketReceiver https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala#L51 Exposing the BlockGenerator interface has made it harder to improve the receiving process. The API of NetworkReceiver (which was not a very stable API anyways) needs to be change if we are to ensure future stability. Additionally, the functions like streamingContext.socketStream that create input streams, return DStream objects. That makes it hard to expose functionality (say, rate limits) unique to input dstreams. They should return InputDStream or NetworkInputDStream. This is still not yet implemented. This PR is blocked on the graceful shutdown PR #247 Author: Tathagata Das <tathagata.das1565@gmail.com> Closes#300 from tdas/network-receiver-api and squashes the following commits: ea27b38 [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into network-receiver-api 3a4777c [Tathagata Das] Renamed NetworkInputDStream to ReceiverInputDStream, and ActorReceiver related stuff. 838dd39 [Tathagata Das] Added more events to the StreamingListener to report errors and stopped receivers. a75c7a6 [Tathagata Das] Address some PR comments and fixed other issues. 91bfa72 [Tathagata Das] Fixed bugs. 8533094 [Tathagata Das] Scala style fixes. 028bde6 [Tathagata Das] Further refactored receiver to allow restarting of a receiver. 43f5290 [Tathagata Das] Made functions that create input streams return InputDStream and NetworkInputDStream, for both Scala and Java. 2c94579 [Tathagata Das] Fixed graceful shutdown by removing interrupts on receiving thread. 9e37a0b [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into network-receiver-api 3223e95 [Tathagata Das] Refactored the code that runs the NetworkReceiver into further classes and traits to make them more testable. a36cc48 [Tathagata Das] Refactored the NetworkReceiver API for future stability. (cherry picked from commit 04c37b6) Signed-off-by: Patrick Wendell <pwendell@gmail.com>

…eam API [WIP] The current Network Receiver API makes it slightly complicated to right a new receiver as one needs to create an instance of BlockGenerator as shown in SocketReceiver https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala#L51 Exposing the BlockGenerator interface has made it harder to improve the receiving process. The API of NetworkReceiver (which was not a very stable API anyways) needs to be change if we are to ensure future stability. Additionally, the functions like streamingContext.socketStream that create input streams, return DStream objects. That makes it hard to expose functionality (say, rate limits) unique to input dstreams. They should return InputDStream or NetworkInputDStream. This is still not yet implemented. This PR is blocked on the graceful shutdown PR apache#247 Author: Tathagata Das <tathagata.das1565@gmail.com> Closesapache#300 from tdas/network-receiver-api and squashes the following commits: ea27b38 [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into network-receiver-api 3a4777c [Tathagata Das] Renamed NetworkInputDStream to ReceiverInputDStream, and ActorReceiver related stuff. 838dd39 [Tathagata Das] Added more events to the StreamingListener to report errors and stopped receivers. a75c7a6 [Tathagata Das] Address some PR comments and fixed other issues. 91bfa72 [Tathagata Das] Fixed bugs. 8533094 [Tathagata Das] Scala style fixes. 028bde6 [Tathagata Das] Further refactored receiver to allow restarting of a receiver. 43f5290 [Tathagata Das] Made functions that create input streams return InputDStream and NetworkInputDStream, for both Scala and Java. 2c94579 [Tathagata Das] Fixed graceful shutdown by removing interrupts on receiving thread. 9e37a0b [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into network-receiver-api 3223e95 [Tathagata Das] Refactored the code that runs the NetworkReceiver into further classes and traits to make them more testable. a36cc48 [Tathagata Das] Refactored the NetworkReceiver API for future stability.

Use theopenlab/ansible instead of ansible/ansible

…error during job running (apache#300)

tdas added 2 commits April 1, 2014 16:26

Refactored the NetworkReceiver API for future stability.
a36cc48

Refactored the code that runs the NetworkReceiver into further classe…
3223e95
…s and traits to make them more testable.

tdas mentioned this pull request Apr 3, 2014
[SPARK-1331] Added graceful shutdown to Spark Streaming #247
Closed

tdas mentioned this pull request Apr 14, 2014
SPARK-1478 #405
Closed

tdas added 4 commits April 14, 2014 19:10

Fixed graceful shutdown by removing interrupts on receiving thread.
2c94579

Made functions that create input streams return InputDStream and Netw…
43f5290
…orkInputDStream, for both Scala and Java.

Further refactored receiver to allow restarting of a receiver.
028bde6

Scala style fixes.
8533094

Merge remote-tracking branch 'apache-github/master' into network-rece…
ea27b38
…iver-api

asfgit closed this in 04c37b6Apr 22, 2014

tdas mentioned this pull request Apr 28, 2014
SPARK-1478: Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915 #566
Closed

bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
Merge pull requestapache#300from theopenlab/openlab-ansible
8256cdd
Use theopenlab/ansible instead of ansible/ansible

wangyum mentioned this pull request May 31, 2020
[SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form #28575
Closed

arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
MapR [SPARK-266] Spark jobs can't finish correctly, when there is an …
fcef2bb
…error during job running (apache#300)

Uh oh!

Conversation

tdas commented Apr 2, 2014

Uh oh!

AmplabJenkins commented Apr 2, 2014

Uh oh!

AmplabJenkins commented Apr 2, 2014

Uh oh!

AmplabJenkins commented Apr 2, 2014

Uh oh!

AmplabJenkins commented Apr 2, 2014

Uh oh!

tdas commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

AmplabJenkins commented Apr 16, 2014

Uh oh!

tdas commented Apr 16, 2014

Uh oh!

tdas commented Apr 16, 2014

Uh oh!

rxin commented Apr 17, 2014

Uh oh!

AmplabJenkins commented Apr 17, 2014

Uh oh!

AmplabJenkins commented Apr 17, 2014

Uh oh!

AmplabJenkins commented Apr 17, 2014

Uh oh!

AmplabJenkins commented Apr 17, 2014

Uh oh!

tdas commented Apr 17, 2014

Uh oh!

AmplabJenkins commented Apr 19, 2014

Uh oh!

AmplabJenkins commented Apr 19, 2014

Uh oh!

AmplabJenkins commented Apr 19, 2014

Uh oh!

AmplabJenkins commented Apr 19, 2014

Uh oh!

tdas commented Apr 21, 2014

Uh oh!

tdas commented Apr 21, 2014

Uh oh!

pwendell commented Apr 21, 2014

Uh oh!

pwendell commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

AmplabJenkins commented Apr 21, 2014

Uh oh!

tdas commented Apr 21, 2014

Uh oh!