[SPARK-3479] [Build] Report failed test category by nchammas · Pull Request #2606 · apache/spark

nchammas · 2014-09-30T22:40:37Z

This PR allows SparkQA (i.e. Jenkins) to report in its posts to GitHub what category of test failed, if one can be determined.

The failure categories are:

general failure
RAT checks failed
Scala style checks failed
Python style checks failed
Build failed
Spark unit tests failed
PySpark unit tests failed
MiMa checks failed

This PR also fixes the diffing logic used to determine if a patch introduces new classes.

nchammas · 2014-09-30T22:41:12Z

I'm gonna post several test commits here to try out the various failure modes and confirm that they are reported correctly.

SparkQA · 2014-09-30T22:49:32Z

QA tests have started for PR 2606 at commit 935f8c0.

This patch merges cleanly.

SparkQA · 2014-09-30T22:49:41Z

QA tests have finished for PR 2606 at commit 935f8c0.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-09-30T22:49:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21072/

AmplabJenkins · 2014-09-30T22:52:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21070/

AmplabJenkins · 2014-10-01T02:02:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21080/

nchammas · 2014-10-01T02:11:11Z

Jenkins failed to checkout the patch.

Jenkins, retest this please.

nchammas · 2014-10-01T02:37:51Z

Jenkins, retest this please.

AmplabJenkins · 2014-10-01T02:52:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21081/

nchammas · 2014-10-01T02:54:52Z

Jenkins, retest this please.

SparkQA · 2014-10-01T02:59:53Z

QA tests have started for PR 2606 at commit d5624ac.

This patch merges cleanly.

SparkQA · 2014-10-01T03:01:02Z

QA tests have finished for PR 2606 at commit d5624ac.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PStatsParam(AccumulatorParam):

AmplabJenkins · 2014-10-01T03:01:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21083/

AmplabJenkins · 2014-10-01T03:27:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21084/

nchammas · 2014-10-01T03:28:27Z

Grr, Jenkins flakiness checking out stuff from git...

Oi Jenkins! Retest this please.

nchammas · 2014-10-01T03:35:40Z

Jenkins, retest this please.

SparkQA · 2014-10-01T03:39:35Z

QA tests have started for PR 2606 at commit 4f5f338.

This patch merges cleanly.

SparkQA · 2014-10-01T03:39:46Z

QA tests have finished for PR 2606 at commit 4f5f338.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PStatsParam(AccumulatorParam):

AmplabJenkins · 2014-10-01T03:39:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21085/

SparkQA · 2014-10-01T03:54:32Z

QA tests have started for PR 2606 at commit 7289777.

This patch merges cleanly.

SparkQA · 2014-10-01T04:09:32Z

QA tests have started for PR 2606 at commit 833e064.

This patch merges cleanly.

SparkQA · 2014-10-01T04:45:58Z

QA tests have finished for PR 2606 at commit 7289777.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-01T04:46:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21086/

SparkQA · 2014-10-01T05:00:40Z

QA tests have finished for PR 2606 at commit 833e064.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-01T05:00:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21087/

SparkQA · 2014-10-01T14:48:58Z

QA tests have started for PR 2606 at commit 1fea6ce.

This patch merges cleanly.

SparkQA · 2014-10-01T14:55:10Z

QA tests have finished for PR 2606 at commit 1fea6ce.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class RDDD[T: ClassTag](

AmplabJenkins · 2014-10-01T14:55:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21116/

SparkQA · 2014-10-01T15:29:40Z

QA tests have started for PR 2606 at commit 28ffbab.

This patch merges cleanly.

SparkQA · 2014-10-01T19:35:18Z

QA tests have finished for PR 2606 at commit f137249.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Dummy()

AmplabJenkins · 2014-10-01T19:35:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21143/

nchammas · 2014-10-01T19:36:57Z

Hmm, I expected a failure in the PySpark doctests, not here.

nchammas · 2014-10-01T19:37:03Z

Jenkins, retest this please.

AmplabJenkins · 2014-10-01T19:47:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21147/

nchammas · 2014-10-01T19:47:38Z

Another checkout timeout.

nchammas · 2014-10-01T19:47:43Z

Jenkins, retest this please.

SparkQA · 2014-10-01T19:54:51Z

QA tests have started for PR 2606 at commit f137249.

This patch merges cleanly.

SparkQA · 2014-10-01T20:46:06Z

QA tests have finished for PR 2606 at commit f137249.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Dummy()

AmplabJenkins · 2014-10-01T20:46:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21152/

SparkQA · 2014-10-01T20:55:03Z

QA tests have started for PR 2606 at commit d67df03.

This patch merges cleanly.

SparkQA · 2014-10-01T22:08:00Z

QA tests have finished for PR 2606 at commit d67df03.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-01T22:08:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21154/

nchammas · 2014-10-02T01:02:01Z

cc @pwendell This PR is ready for review.

Here are examples of the messages posted when:

nchammas · 2014-10-02T01:03:45Z

We can do a valid diff regardless of the merge-ability of the patch, so I moved this out of the if block.

JoshRosen · 2014-10-02T20:20:42Z

This looks great! Thanks for adding the extra comments to explain the $sha and $ghprbActualCommit variables, since they can be kind of confusing.

nchammas · 2014-10-06T21:01:29Z

@pwendell @JoshRosen Is this PR ready to be merged in?

JoshRosen · 2014-10-06T21:18:45Z

Sure, this looks fine to me so I'm going to merge it now. Thanks!

…h modern GitHub Actions grouping ### What changes were proposed in this pull request? Remove the old `CURRENT_BLOCK` and `ERROR_CODES` testing infrastructure and replace it with an updated `titled_block` context manager that also groups log output on GitHub Actions. Move section titles from the various helper functions into titled blocks in `main()`. Exit codes are no longer driven by the old blocks. If the tests fail, the script exits with a non-zero code, usually `1`. Make sure that titled blocks cannot be nested, and that all large blocks of output have a suitable title block at the top level `main()` function. That means I moved the following checks and calls up to `main()`: - `run_java_style_checks` - `build_spark_unidoc_sbt` - `SKIP_PACKAGING` flag check - `SKIP_UNIDOC` flag check Fold `group_in_github_actions` into the new `titled_block`. Remove `build_spark_documentation`, which is not used by anything. ### Why are the changes needed? Amusingly, I added this testing infrastructure 12 years ago in #2606 (later refactored into Python in #5694). It was built for Jenkins, which we do not use anymore. The block and error codes are not used by any part of our current test infrastructure on GitHub Actions. The new `titled_block` context manager takes care of printing block titles as before, and also subsumes the log grouping behavior from `group_in_github_actions` (introduced in #54524) since it is useful for more than just sbt. We have a [separate job for building the docs][docs], so I assume we don't want to revive the currently-dead `build_spark_documentation`. [docs]: https://github.com/apache/spark/blob/2e83cd8444d787203ed2e594f78a1c54bfcaff42/.github/workflows/build_and_test.yml#L1218-L1240 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI for this PR. Some illustrative screenshots of the new log groups: <img width="375" src="https://github.com/user-attachments/assets/80da8dff-15a5-4569-b224-580d7bf029de" /> <img width="286" src="https://github.com/user-attachments/assets/32469a0c-77bf-44f9-ac16-c3a12662fcc7" /> The purple debug lines are from my fork and are not related to this PR. ### Was this patch authored or co-authored using generative AI tooling? I wrote this with assistance from GitHub Copilot. Closes#57770 from nchammas/run-tests-titled-block. Authored-by: Nicholas Chammas <nicholas.chammas@gmail.com> Signed-off-by: Nicholas Chammas <nicholas.chammas@gmail.com>

…h modern GitHub Actions grouping ### What changes were proposed in this pull request? Remove the old `CURRENT_BLOCK` and `ERROR_CODES` testing infrastructure and replace it with an updated `titled_block` context manager that also groups log output on GitHub Actions. Move section titles from the various helper functions into titled blocks in `main()`. Exit codes are no longer driven by the old blocks. If the tests fail, the script exits with a non-zero code, usually `1`. Make sure that titled blocks cannot be nested, and that all large blocks of output have a suitable title block at the top level `main()` function. That means I moved the following checks and calls up to `main()`: - `run_java_style_checks` - `build_spark_unidoc_sbt` - `SKIP_PACKAGING` flag check - `SKIP_UNIDOC` flag check Fold `group_in_github_actions` into the new `titled_block`. Remove `build_spark_documentation`, which is not used by anything. ### Why are the changes needed? Amusingly, I added this testing infrastructure 12 years ago in #2606 (later refactored into Python in #5694). It was built for Jenkins, which we do not use anymore. The block and error codes are not used by any part of our current test infrastructure on GitHub Actions. The new `titled_block` context manager takes care of printing block titles as before, and also subsumes the log grouping behavior from `group_in_github_actions` (introduced in #54524) since it is useful for more than just sbt. We have a [separate job for building the docs][docs], so I assume we don't want to revive the currently-dead `build_spark_documentation`. [docs]: https://github.com/apache/spark/blob/2e83cd8444d787203ed2e594f78a1c54bfcaff42/.github/workflows/build_and_test.yml#L1218-L1240 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI for this PR. Some illustrative screenshots of the new log groups: <img width="375" src="https://github.com/user-attachments/assets/80da8dff-15a5-4569-b224-580d7bf029de" /> <img width="286" src="https://github.com/user-attachments/assets/32469a0c-77bf-44f9-ac16-c3a12662fcc7" /> The purple debug lines are from my fork and are not related to this PR. ### Was this patch authored or co-authored using generative AI tooling? I wrote this with assistance from GitHub Copilot. Closes#57770 from nchammas/run-tests-titled-block. Authored-by: Nicholas Chammas <nicholas.chammas@gmail.com> Signed-off-by: Nicholas Chammas <nicholas.chammas@gmail.com> (cherry picked from commit cef67d7) Signed-off-by: Nicholas Chammas <nicholas.chammas@gmail.com>

nchammas force-pushed the report-failed-test-category branch from 0f6dcd3 to cc243c9Compare September 30, 2014 22:41

report what test category failed
d67df03

nchammas force-pushed the report-failed-test-category branch from f137249 to d67df03Compare October 1, 2014 20:48

nchammas changed the title ~~[SPARK-3479] [Build] [WIP] Report failed test category~~[SPARK-3479] [Build] Report failed test categoryOct 2, 2014

nchammas reviewed Oct 2, 2014
View reviewed changes

nchammas mentioned this pull request Oct 2, 2014
[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states #2339
Closed

asfgit closed this in 69c3f44Oct 6, 2014

JoshRosen mentioned this pull request Oct 6, 2014
Show which tests failed databricks/spark-pr-dashboard#24
Closed

nchammas deleted the report-failed-test-category branch October 8, 2014 01:05

nchammas mentioned this pull request Aug 4, 2026
[SPARK-58573][INFRA][BUILD] Replace defunct Jenkins test grouping with modern GitHub Actions grouping #57770
Closed

Uh oh!

Conversation

nchammas commented Sep 30, 2014

Uh oh!

nchammas commented Sep 30, 2014

Uh oh!

SparkQA commented Sep 30, 2014

Uh oh!

SparkQA commented Sep 30, 2014

Uh oh!

AmplabJenkins commented Sep 30, 2014

Uh oh!

AmplabJenkins commented Sep 30, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

AmplabJenkins commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

nchammas commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 1, 2014

Uh oh!