[SPARK-4026][Streaming] Write ahead log management

[tdas]

As part of the effort to avoid data loss on Spark Streaming driver failure, we want to implement a write ahead log that can write received data to HDFS. This allows the received data to be persist across driver failures. So when the streaming driver is restarted, it can find and reprocess all the data that were received but not processed.

This was primarily implemented by @harishreedharan. This is still WIP, as he is going to improve the unitests by using HDFS mini cluster.

[tdas]

Please review this @JoshRosen

[SparkQA]

QA tests have started for PR 2882 at commit 5182ffb.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2882 at commit 5182ffb.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LogInfo(startTime: Long, endTime: Long, path: String)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22002/
Test FAILed.

[SparkQA]

QA tests have started for PR 2882 at commit 4ab602a.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2882 at commit 4ab602a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LogInfo(startTime: Long, endTime: Long, path: String)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22006/
Test PASSed.

[tdas]

WIP: this file is going to be updated by @harishreedharan to get rid of the local file customizations.

[JoshRosen]

Ah, that makes sense. I guess you can still use the HDFS API to write to local files for testing purposes.

[harishreedharan]

Yep. And for all tests, we are just going to use Hadoop Minicluster anyway.

[JoshRosen]

Nice Scala one-liner :)

Why do we need this reflection, though? Is this necessary to support multiple Hadoop versions? If so, could you add a one-line comment to explain this?

[harishreedharan]

Actually we do, since Spark supports Hadoop 1 to Hadoop 2.5.0 right now. In Hadoop 1.x, the "sync" method did the same thing hflush does in 2.5.0 - so in short we do.

[tdas]

Credit goes to Colin McCabe who wrote this line.
https://github.com/apache/spark/blame/master/core/src/main/scala/org/apache/spark/util/FileLogger.scala#L106
Stole from there.

[SparkQA]

QA tests have finished for PR 2882 at commit 3881706.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LogInfo(startTime: Long, endTime: Long, path: String)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22067/
Test PASSed.

[SparkQA]

QA tests have started for PR 2882 at commit 9514dc8.

• This patch merges cleanly.

[tdas]

@JoshRosen
@harishreedharan addressed all your comments, and also simplified the writer code
I did some further cleanups, and also added two new unit tests that test the writer and manager with corrupted writes.

[SparkQA]

QA tests have finished for PR 2882 at commit 9514dc8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LogInfo(startTime: Long, endTime: Long, path: String)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22068/
Test FAILed.

[SparkQA]

QA tests have started for PR 2882 at commit d29fddd.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2882 at commit d29fddd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LogInfo(startTime: Long, endTime: Long, path: String)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22075/
Test PASSed.

[harishreedharan]

Yay, finally!

[tdas]

@JoshRosen whenever you get a chance. :)

[JoshRosen]

It looks like this comment is no longer relevant, or perhaps like it should be moved somewhere else?

[tdas]

Moved.

[JoshRosen]

This looks good to me!

[tdas]

Alright, thanks! I will merge when this last set of changes gets through jenkins.

[SparkQA]

QA tests have started for PR 2882 at commit e4bee20.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2882 at commit e4bee20.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LogInfo(startTime: Long, endTime: Long, path: String)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22103/
Test PASSed.

[harishreedharan]

Let's merge this for now. I will try and find out more about the getFileSystem thread-safety without doAs (which is what we support anyway)

[harishreedharan]

Talked to @cmccabe who says we should not worry about the thread-safety. If at all there was an issue, it was in too old a version which we need not worry about. Let's merge this!

[tdas]

Cool! Thanks for check with @cmccabe. Merging this.