[SQL][WIP] Refined Thrift server test suite by liancheng · Pull Request #2214 · apache/spark

liancheng · 2014-08-30T09:08:14Z

NOTE

This PR is not ready to be merged yet due to some issue only reproducible with Jenkins build triggered by GitHub.

This PR fixes two issues of HiveThriftServer2Suite and brings 1 enhancement:

Although metastore, warehouse directories and listening port are randomly chosen, all test cases share the same configuration. Due to parallel test execution, one of the two test case is doomed to fail
We caught any exceptions thrown from a test case and print diagnosis information, but forgot to re-throw the exception...
When the forked server process ends prematurely (e.g., fails to start), the serverRunning promise is completed with a failure, preventing the test code to keep waiting until timeout.

So, embarrassingly, this test suite was failing continuously for several days but no one had ever noticed it... Fortunately no bugs in the production code were covered under the hood.

liancheng · 2014-08-30T18:58:49Z

test this please

SparkQA · 2014-08-30T19:04:06Z

QA tests have started for PR 2214 at commit 983d030.

This patch merges cleanly.

SparkQA · 2014-08-30T20:20:08Z

QA tests have finished for PR 2214 at commit 983d030.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-09-03T05:07:13Z

ok to test

liancheng · 2014-09-03T07:40:45Z

test this please

SparkQA · 2014-09-03T07:44:32Z

QA tests have started for PR 2214 at commit 94a83ba.

This patch merges cleanly.

liancheng · 2014-09-03T07:59:47Z

The test failures only occur when running on Jenkins, I couldn't reproduce it either locally or on Jenkins server node. May fire some debugging commits later to investigate this issue.

SparkQA · 2014-09-03T09:27:10Z

QA tests have finished for PR 2214 at commit 94a83ba.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-09-03T22:05:10Z

retest this please

liancheng · 2014-09-03T23:14:12Z

ok to test

SparkQA · 2014-09-03T23:19:27Z

QA tests have started for PR 2214 at commit 1931e96.

This patch merges cleanly.

SparkQA · 2014-09-03T23:24:53Z

QA tests have finished for PR 2214 at commit 1931e96.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: BlockManagerId, maxMem: Long)
- case class SparkListenerBlockManagerRemoved(time: Long, blockManagerId: BlockManagerId)
- case class SparkListenerApplicationStart(appName: String, appId: Option[String], time: Long,

liancheng · 2014-09-03T23:47:20Z

Hmm, a summary about the failure pattern of HiveThriftServer2Suite:

Running locally with
- hive-thriftserver/test-only *.*Suite
  PASS
- test
  PASS
SSH to Jenkins node, and then
- run with sudo and hive-thriftserver/test-only *.*Suite
  PASS
- run with sudo and test
  PASS
Jenkins build triggered by GitHub with
- modified dev/run-tests script that only hive-thriftserver/test-only *.*Suite
  PASS
- normal dev/run-tests script
  FAIL
  ... even if HiveThriftServer2 is the first test suite that got executed. sbin/start-thriftserver.sh script is called, since we can see output of compute-classpath.sh from the log, but it seems that the JVM process fails to start.

SparkQA · 2014-09-03T23:59:24Z

QA tests have started for PR 2214 at commit 1185d79.

This patch merges cleanly.

SparkQA · 2014-09-04T00:14:49Z

QA tests have finished for PR 2214 at commit 1185d79.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-04T04:04:25Z

QA tests have started for PR 2214 at commit c537e37.

This patch merges cleanly.

liancheng · 2014-09-04T05:15:25Z

add to whitelist

liancheng · 2014-09-04T05:15:47Z

ok to test

SparkQA · 2014-09-04T06:04:26Z

Tests timed out after a configured wait of 120m.

SparkQA · 2014-09-05T23:53:21Z

QA tests have started for PR 2214 at commit a1ad308.

This patch merges cleanly.

SparkQA · 2014-09-06T01:53:21Z

Tests timed out after a configured wait of 120m.

marmbrus · 2014-09-09T02:49:01Z

test this please

marmbrus · 2014-09-09T02:49:16Z

@liancheng are we still debugging issues here? or just waiting for it to pass?

SparkQA · 2014-09-09T03:49:09Z

QA tests have started for PR 2214 at commit a1ad308.

This patch merges cleanly.

liancheng · 2014-09-09T04:08:46Z

@marmbrus not ready yet, was not able to debug it since Jenkins was quite crazy these days. I'll remove the WIP tag once it's ready.

SparkQA · 2014-09-09T05:36:08Z

QA tests have finished for PR 2214 at commit a1ad308.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-11T00:49:06Z

QA tests have started for PR 2214 at commit a1ad308.

This patch merges cleanly.

SparkQA · 2014-09-11T00:53:23Z

QA tests have started for PR 2214 at commit 23d96f1.

This patch merges cleanly.

SparkQA · 2014-09-11T02:25:15Z

QA tests have finished for PR 2214 at commit a1ad308.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-11T02:47:24Z

QA tests have finished for PR 2214 at commit 23d96f1.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-14T02:59:39Z

QA tests have started for PR 2214 at commit 23d96f1.

This patch merges cleanly.

SparkQA · 2014-09-14T04:10:53Z

QA tests have finished for PR 2214 at commit 23d96f1.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-10-08T01:35:45Z

#2675 supersedes this one, closing.

As scwf pointed out, `HiveThriftServer2Suite` isn't effective anymore after the Thrift server was made a daemon. On the other hand, these test suites were known flaky, PR #2214 tried to fix them but failed because of unknown Jenkins build error. This PR fixes both sets of issues. In this PR, instead of watching `start-thriftserver.sh` output, the test code start a `tail` process to watch the log file. A `Thread.sleep` has to be introduced because the `kill` command used in `stop-thriftserver.sh` is not synchronous. As for the root cause of the mysterious Jenkins build failure. Please refer to [this comment](#2675 (comment)) below for details. ---- (Copied from PR description of #2214) This PR fixes two issues of `HiveThriftServer2Suite` and brings 1 enhancement: 1. Although metastore, warehouse directories and listening port are randomly chosen, all test cases share the same configuration. Due to parallel test execution, one of the two test case is doomed to fail 2. We caught any exceptions thrown from a test case and print diagnosis information, but forgot to re-throw the exception... 3. When the forked server process ends prematurely (e.g., fails to start), the `serverRunning` promise is completed with a failure, preventing the test code to keep waiting until timeout. So, embarrassingly, this test suite was failing continuously for several days but no one had ever noticed it... Fortunately no bugs in the production code were covered under the hood. Author: Cheng Lian <lian.cs.zju@gmail.com> Author: wangfei <wangfei1@huawei.com> Closes#2675 from liancheng/fix-thriftserver-tests and squashes the following commits: 1c384b7 [Cheng Lian] Minor code cleanup, restore the logging level hack in TestHive.scala 7805c33 [wangfei] reset SPARK_TESTING to avoid loading Log4J configurations in testing class paths af2b5a9 [Cheng Lian] Removes log level hacks from TestHiveContext d116405 [wangfei] make sure that log4j level is INFO ee92a82 [Cheng Lian] Relaxes timeout 7fd6757 [Cheng Lian] Fixes test suites in hive-thriftserver

As scwf pointed out, `HiveThriftServer2Suite` isn't effective anymore after the Thrift server was made a daemon. On the other hand, these test suites were known flaky, PR apache#2214 tried to fix them but failed because of unknown Jenkins build error. This PR fixes both sets of issues. In this PR, instead of watching `start-thriftserver.sh` output, the test code start a `tail` process to watch the log file. A `Thread.sleep` has to be introduced because the `kill` command used in `stop-thriftserver.sh` is not synchronous. As for the root cause of the mysterious Jenkins build failure. Please refer to [this comment](apache#2675 (comment)) below for details. ---- (Copied from PR description of apache#2214) This PR fixes two issues of `HiveThriftServer2Suite` and brings 1 enhancement: 1. Although metastore, warehouse directories and listening port are randomly chosen, all test cases share the same configuration. Due to parallel test execution, one of the two test case is doomed to fail 2. We caught any exceptions thrown from a test case and print diagnosis information, but forgot to re-throw the exception... 3. When the forked server process ends prematurely (e.g., fails to start), the `serverRunning` promise is completed with a failure, preventing the test code to keep waiting until timeout. So, embarrassingly, this test suite was failing continuously for several days but no one had ever noticed it... Fortunately no bugs in the production code were covered under the hood. Author: Cheng Lian <lian.cs.zju@gmail.com> Author: wangfei <wangfei1@huawei.com> Closesapache#2675 from liancheng/fix-thriftserver-tests and squashes the following commits: 1c384b7 [Cheng Lian] Minor code cleanup, restore the logging level hack in TestHive.scala 7805c33 [wangfei] reset SPARK_TESTING to avoid loading Log4J configurations in testing class paths af2b5a9 [Cheng Lian] Removes log level hacks from TestHiveContext d116405 [wangfei] make sure that log4j level is INFO ee92a82 [Cheng Lian] Relaxes timeout 7fd6757 [Cheng Lian] Fixes test suites in hive-thriftserver Conflicts: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala

liancheng force-pushed the refine-thrift-test branch from 983d030 to 94a83baCompare September 3, 2014 03:37

liancheng changed the title ~~[SQL] Refined Thrift server test suite~~[SQL][WIP] Refined Thrift server test suiteSep 4, 2014

liancheng force-pushed the refine-thrift-test branch from 1185d79 to c537e37Compare September 4, 2014 03:58

Refined Thrift server test suite
389e181

liancheng added 7 commits September 10, 2014 17:46

Similar fixes for CliSuite
83d88c7

Debugging commit to diagnose Jenkins build
0fd8f1a

Enables all other checks
201fddd

Revert dev/run-tests changes
2a934c0

Added debugging output in bin/spark-class
2c0546d

Revert dev/run-tests changes
d433fea

Revert bin/spark-class debugging changes
23d96f1

liancheng force-pushed the refine-thrift-test branch from a1ad308 to 23d96f1Compare September 11, 2014 00:47

This was referenced Oct 6, 2014
[SPARK-3809][SQL]fix HiveThriftServer2Suite to make it work correctly #2671
Closed
[SPARK-3809][SQL] Fixes test suites in hive-thriftserver #2675
Closed

liancheng closed this Oct 8, 2014

liancheng deleted the refine-thrift-test branch October 8, 2014 01:35

liancheng mentioned this pull request Nov 5, 2014
[SPARK-3971][SQL] Backport #2843 to branch-1.1 #3113
Closed

Uh oh!

Conversation

liancheng commented Aug 30, 2014

Uh oh!

liancheng commented Aug 30, 2014

Uh oh!

SparkQA commented Aug 30, 2014

Uh oh!

SparkQA commented Aug 30, 2014

Uh oh!

marmbrus commented Sep 3, 2014

Uh oh!

liancheng commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

liancheng commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

liancheng commented Sep 3, 2014

Uh oh!

liancheng commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

liancheng commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 4, 2014

Uh oh!

SparkQA commented Sep 4, 2014

Uh oh!

liancheng commented Sep 4, 2014

Uh oh!

liancheng commented Sep 4, 2014

Uh oh!

SparkQA commented Sep 4, 2014

Uh oh!

SparkQA commented Sep 5, 2014

Uh oh!

SparkQA commented Sep 6, 2014

Uh oh!

marmbrus commented Sep 9, 2014

Uh oh!

marmbrus commented Sep 9, 2014

Uh oh!

SparkQA commented Sep 9, 2014

Uh oh!

liancheng commented Sep 9, 2014

Uh oh!

SparkQA commented Sep 9, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 14, 2014

Uh oh!

SparkQA commented Sep 14, 2014

Uh oh!

liancheng commented Oct 8, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants