Skip to content

[Hot Fix][Streaming] Fix WriteAheadLogBackedBlockRDDSuite - #3701

Closed
scwf wants to merge 1 commit into
apache:masterfrom
scwf:patch-10
Closed

[Hot Fix][Streaming] Fix WriteAheadLogBackedBlockRDDSuite#3701
scwf wants to merge 1 commit into
apache:masterfrom
scwf:patch-10

Conversation

@scwf

@scwfscwf commented Dec 15, 2014

Copy link
Copy Markdown
Contributor

WriteAheadLogBackedBlockRDDSuite failed due to the same random file name(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24456/consoleFull)

/tmp/1418645532799-0/??????????
[info] - Read data available only in write ahead log, and test storing in block manager *** FAILED *** (2 milliseconds)

Using another way to produce random file name(http://alvinalexander.com/scala/creating-random-strings-in-scala).

@scwf

scwf commented Dec 15, 2014

Copy link
Copy Markdown
ContributorAuthor

link to #3687

@SparkQA

SparkQA commented Dec 15, 2014

Copy link
Copy Markdown

Test build #24458 has started for PR 3701 at commit b7f8955.

  • This patch merges cleanly.

@scwfscwf mentioned this pull request Dec 15, 2014
Closed
@srowen

srowen commented Dec 15, 2014

Copy link
Copy Markdown
Member

This is for SPARK-4826 right? can you explain the fix? is it that reserved characters like / or non-ASCII chars were part of the file name?

@srowensrowen mentioned this pull request Dec 15, 2014
Closed
@scwf

scwf commented Dec 15, 2014

Copy link
Copy Markdown
ContributorAuthor

Yes, it is. I have not noticed that PR. Based on my test, this failure is caused by Random.nextString(10), it produce the same string each time when test(a debug link on jenkins: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24456/consoleFull) for unkonw reason. I am not sure this is caused by bugs of Random.nextString or race condition

@scwfscwf changed the title [Hot Fix] Fix WriteAheadLogBackedBlockRDDSuite[Hot Fix][Streaming] Fix WriteAheadLogBackedBlockRDDSuiteDec 15, 2014
@SparkQA

SparkQA commented Dec 15, 2014

Copy link
Copy Markdown

Test build #24458 has finished for PR 3701 at commit b7f8955.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Dec 15, 2014

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24458/
Test PASSed.

@JoshRosenJoshRosen mentioned this pull request Dec 15, 2014
Closed
@JoshRosen

JoshRosen commented Dec 15, 2014

Copy link
Copy Markdown
Contributor

I've opened a new PR with what I think is is a more direct approach to fixing this issue: #3704. Please take a look and let me know what you think.

asfgit pushed a commit that referenced this pull request Dec 15, 2014
@JoshRosen
[SPARK-4826] Fix generation of temp file names in WAL tests
c5a9ae6
This PR should fix SPARK-4826, an issue where a bug in how we generate temp. file names was causing spurious test failures in the write ahead log suites.
Closes#3695.
Closes#3701.
Author: Josh Rosen <joshrosen@databricks.com>
Closes#3704 from JoshRosen/SPARK-4826 and squashes the following commits:
f2307f5 [Josh Rosen] Use Spark Utils class for directory creation/deletion
a693ddb [Josh Rosen] remove unused Random import
b275e41 [Josh Rosen] Move creation of temp. dir to beforeEach/afterEach.
9362919 [Josh Rosen] [SPARK-4826] Fix bug in generation of temp file names. in WAL suites.
86c1944 [Josh Rosen] Revert "HOTFIX: Disabling failing block manager test"
(cherry picked from commit f6b8591)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
@scwf
scwf deleted the patch-10 branch December 17, 2014 15:39
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@scwf@SparkQA@srowen@AmplabJenkins@JoshRosen