[SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite by jiangxb1987 · Pull Request #28584 · apache/spark

jiangxb1987 · 2020-05-19T23:16:39Z

What changes were proposed in this pull request?

To wait until all the executors have started before submitting any job. This could avoid the flakiness caused by waiting for executors coming up.

How was this patch tested?

Existing tests.

dongjoon-hyun · 2020-05-19T23:55:59Z

Thank you so much, @jiangxb1987 .

Ngone51

LGTM

SparkQA · 2020-05-20T01:45:13Z

Test build #122856 has finished for PR 28584 at commit b08a272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2020-05-20T02:05:09Z

"SPARK-31485: barrier stage should fail if only partial tasks are launched" This test case is flaky, @Ngone51 can you take a look?

Ngone51 · 2020-05-20T02:08:40Z

sure, thanks!

SparkQA · 2020-05-20T02:22:00Z

Test build #122860 has finished for PR 28584 at commit b08a272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-20T02:22:42Z

Test build #122864 has finished for PR 28584 at commit b08a272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-05-20T02:28:42Z

retest this please.

SparkQA · 2020-05-20T02:31:12Z

Test build #122862 has finished for PR 28584 at commit b08a272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-20T05:13:50Z

Test build #122865 has finished for PR 28584 at commit b08a272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

WeichenXu123 · 2020-05-21T03:38:31Z

Also fix this test like:

 test("share messages with allGather() call") {
val conf = new SparkConf()
.setMaster("local-cluster[4, 1, 1024]")
.setAppName("test-cluster")
sc = new SparkContext(conf)
val rdd = sc.makeRDD(1 to 4, 4)
val rdd2 = rdd.barrier().mapPartitions { it =>
val context = BarrierTaskContext.get()
// Sleep for a random time before global sync.
Thread.sleep(Random.nextInt(1000))
// Pass partitionId message in
val message: String = context.partitionId().toString
val messages: Array[String] = context.allGather(message)
Iterator.single(messages.toList)
}
// Take a sorted list of all the partitionId messages
val messages_list = rdd2.collect()
assert (messages_list.length === 4)
for (messages <- messages_list) {
assert (messages === (0 until 4).map(_.toString).toList)
}
}

Ngone51 · 2020-05-21T08:18:57Z

Opened a separate PR #28596 to fix the test @WeichenXu123

dongjoon-hyun · 2020-05-24T20:22:49Z

Retest this please.

dongjoon-hyun · 2020-05-24T20:23:28Z

Since #28596 is merged, shall we restart to proceed this PR? Thanks.

SparkQA · 2020-05-24T22:21:26Z

Test build #123066 has finished for PR 28584 at commit b08a272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-05-26T02:35:20Z

ping @jiangxb1987 Could you please update this?

SparkQA · 2020-05-26T06:50:00Z

Test build #123106 has finished for PR 28584 at commit 83a34c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-05-26T06:56:58Z

retest this please.

SparkQA · 2020-05-26T07:05:01Z

Test build #123110 has finished for PR 28584 at commit 83a34c2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-05-26T13:46:20Z

retest this please.

SparkQA · 2020-05-26T16:51:17Z

Test build #123126 has finished for PR 28584 at commit 83a34c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2020-05-26T17:10:38Z

Retest this please.

jiangxb1987 · 2020-05-26T17:12:42Z

Retest this please.

SparkQA · 2020-05-26T20:03:33Z

Test build #123134 has finished for PR 28584 at commit 83a34c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2020-05-26T20:51:23Z

Retest this please.

jiangxb1987 · 2020-05-26T20:52:14Z

Let's trigger one more time to ensure the suite is no longer flaky.

SparkQA · 2020-05-26T23:52:35Z

Test build #123138 has finished for PR 28584 at commit 83a34c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2020-05-27T18:05:11Z

Retest this please.

SparkQA · 2020-05-27T21:43:33Z

Test build #123199 has finished for PR 28584 at commit 83a34c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? To wait until all the executors have started before submitting any job. This could avoid the flakiness caused by waiting for executors coming up. ### How was this patch tested? Existing tests. Closes#28584 from jiangxb1987/barrierTest. Authored-by: Xingbo Jiang <xingbo.jiang@databricks.com> Signed-off-by: Xingbo Jiang <xingbo.jiang@databricks.com> (cherry picked from commit efe7fd2) Signed-off-by: Xingbo Jiang <xingbo.jiang@databricks.com>

jiangxb1987 · 2020-05-27T23:37:49Z

Thanks, merged to master/3.0 !

### What changes were proposed in this pull request? To wait until all the executors have started before submitting any job. This could avoid the flakiness caused by waiting for executors coming up. ### How was this patch tested? Existing tests. Closesapache#28584 from jiangxb1987/barrierTest. Authored-by: Xingbo Jiang <xingbo.jiang@databricks.com> Signed-off-by: Xingbo Jiang <xingbo.jiang@databricks.com>

jiangxb1987 · 2020-05-28T00:31:40Z

Opened #28658 for branch-3.0

fix
b08a272

probot-autolabelerBot added the CORE label May 19, 2020

Ngone51 approved these changes May 20, 2020
View reviewed changes

seanli-rallyhealth approved these changes May 25, 2020
View reviewed changes

fix
83a34c2

Ngone51 mentioned this pull request May 27, 2020
[SPARK-31837][CORE] Shift to the new highest locality level if there is when recomputeLocality #28656
Closed

jiangxb1987 closed this in efe7fd2May 27, 2020

Uh oh!

Conversation

jiangxb1987 commented May 19, 2020

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun commented May 19, 2020

Uh oh!

Ngone51 left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

jiangxb1987 commented May 20, 2020

Uh oh!

Ngone51 commented May 20, 2020

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

Ngone51 commented May 20, 2020

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

WeichenXu123 commented May 21, 2020

Uh oh!

Ngone51 commented May 21, 2020

Uh oh!

dongjoon-hyun commented May 24, 2020

Uh oh!

dongjoon-hyun commented May 24, 2020

Uh oh!

SparkQA commented May 24, 2020

Uh oh!

Ngone51 commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

Ngone51 commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

Ngone51 commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

jiangxb1987 commented May 26, 2020

Uh oh!

jiangxb1987 commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

jiangxb1987 commented May 26, 2020

Uh oh!

jiangxb1987 commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

jiangxb1987 commented May 27, 2020

Uh oh!

SparkQA commented May 27, 2020

Uh oh!

jiangxb1987 commented May 27, 2020

Uh oh!

jiangxb1987 commented May 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants