[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests by JoshRosen · Pull Request #10461 · apache/spark

JoshRosen · 2015-12-23T23:46:33Z

This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath.

This supplants the checks added in SPARK-4123 / #5093, which are currently disabled due to bugs.

This patch is based on @pwendell's work in #8531.

Closes#8531.

Conflicts: dev/run-tests-jenkins dev/sparktestsupport/modules.py

…atching)

JoshRosen · 2015-12-23T23:47:18Z

/cc @pwendell @srowen @brennonyork @nchammas

JoshRosen · 2015-12-30T08:10:55Z

Uh oh, I'd hold off on merging this pending investigation of issues with the dummy change.

SparkQA · 2015-12-30T09:06:22Z

Test build #48452 has finished for PR 10461 at commit 9932c80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-30T19:22:15Z

Spotted the problem: the original logic for checking should_run_build_tests was wrong: it needs to loop over all test_modules (using any) and the root module needs to have the build tests enabled.

SparkQA · 2015-12-30T19:30:12Z

Test build #48503 has finished for PR 10461 at commit 1454f5f.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-30T19:38:27Z

Woohoo, that worked! However, I'm not a fan of how vanilla diff formats its output, so I'm going to quickly try and see if I can use git diff for this.

SparkQA · 2015-12-30T20:10:14Z

Test build #48505 has finished for PR 10461 at commit 329bdf2.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-30T20:46:25Z

Going to merge this into master and will email the dev list with a summary describing the changes + impact on process.

SparkQA · 2015-12-30T22:04:43Z

Test build #48507 has finished for PR 10461 at commit cc99de6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in #10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closes#10543 from JoshRosen/dep-script-fixes.

…r new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / apache#5093, which are currently disabled due to bugs. This patch is based on pwendell's work in apache#8531. Closesapache#8531. Author: Josh Rosen <joshrosen@databricks.com> Author: Patrick Wendell <patrick@databricks.com> Closesapache#10461 from JoshRosen/SPARK-10359.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in apache#10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closesapache#10543 from JoshRosen/dep-script-fixes.

…r new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / apache#5093, which are currently disabled due to bugs. This patch is based on pwendell's work in apache#8531. Closesapache#8531. Author: Josh Rosen <joshrosen@databricks.com> Author: Patrick Wendell <patrick@databricks.com> Closesapache#10461 from JoshRosen/SPARK-10359.

…sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in apache#10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen <joshrosen@databricks.com> Closesapache#10543 from JoshRosen/dep-script-fixes.

… branch-1.6 This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.6. Author: Josh Rosen <joshrosen@databricks.com> Closes#10680 from JoshRosen/test-deps-16-backport.

…o branch-1.5 This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.5. Author: Josh Rosen <joshrosen@databricks.com> Closes#10679 from JoshRosen/test-deps-15-backport and squashes the following commits: 7633793 [Josh Rosen] Fix merge conflicts. 1d60cb6 [Josh Rosen] [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching 889c355 [Josh Rosen] [SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script 4e48f14 [Josh Rosen] [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests

pwendelland others added 26 commits August 28, 2015 17:18

Adding build test module
9cf442d

Adding new deps file without spark
20ae791

Moving build tests earlier
5670b6a

Adding license header
5b2910b

Adding RAT exclude
0c8153d

Merge remote-tracking branch 'apache/master' into dependency-audits
f4243b9
Conflicts: dev/run-tests-jenkins dev/sparktestsupport/modules.py

Ignore spark itself in deps file
17970f7

Style fix
16adab6

Adding test block
8b973bf

Only include compile dependencies when generating list
d181e3d

Adding Hadoop 1 profile
84cab2c

Adding new hadoop versions
5c8ccf6

Bug fix
fcea19f

Adding new hadoop deps
2f4d3e5

More RAT excludes
37230f0

Testing new dependencies
1d91634

Merge remote-tracking branch 'origin/master' into SPARK-10359
1e3ff47

Refactor to avoid compile in install.
b013514

Don't generate version backup POMs.
8f7007e

Use bash exit traps to ensure proper cleanup
258e4f7

Whoops, missed another -DgenerateBackupPoms=false
e8af274

Support multiple Hadoop profiles.
8507ffd

Roll back test dependency change.
2711252

Fix RAT excludes (CLI RAT doesn't seem to support full path pattern m…
4ad3ae4
…atching)

dev/test-dependencies.sh is part of the build module.
7c86eb7

Use runtime scope (per Sean's comments)
87918d8

JoshRosen mentioned this pull request Dec 23, 2015
[SPARK-4123][Project Infra]: Show new dependencies added in pull requests #5093
Closed

JoshRosen added 2 commits December 23, 2015 15:48

Remove old dependency diffing code.
15a0e68

Remove accidentally-added files
5acdd9a

JoshRosen changed the title ~~[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests~~[SPARK-10359][DONOTMERGE] Enumerate dependencies in a file and diff against it for new pull requestsDec 30, 2015

Changes to test module system
1454f5f

JoshRosen added 3 commits December 30, 2015 11:53

Use git diff for nicer diff formatting.
329bdf2

Minor change in exit trap
367c8ed

Roll back test dependency change.
cc99de6

JoshRosen changed the title ~~[SPARK-10359][DONOTMERGE] Enumerate dependencies in a file and diff against it for new pull requests~~[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requestsDec 30, 2015

asfgit closed this in 27a42c7Dec 30, 2015

JoshRosen deleted the SPARK-10359 branch December 30, 2015 20:50

JoshRosen mentioned this pull request Jan 1, 2016
[SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script #10543
Closed

JoshRosen mentioned this pull request Jan 10, 2016
[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.5 #10679
Closed

JoshRosen mentioned this pull request Jan 10, 2016
[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.6 #10680
Closed

Navigation Menu

Uh oh!

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests - #10461

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests#10461
JoshRosen wants to merge 37 commits into
apache:masterfrom
JoshRosen:SPARK-10359

JoshRosen commented Dec 23, 2015

Uh oh!

JoshRosen commented Dec 23, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

JoshRosen commented Dec 23, 2015

Uh oh!

JoshRosen commented Dec 23, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

JoshRosen commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants