[SPARK-10359] Enumerate Spark's dependencies in a file and diff against it for new pull requests

[pwendell]

DON'T MERGE ME - TESTING ON JENKINS

[pwendell]

Jenkins, test this please.

[SparkQA]

Test build #41816 has finished for PR 8531 at commit 9cf442d.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

Jenkins, retest this please.

[pwendell]

Jenkins, test this please.

[SparkQA]

Test build #41817 has finished for PR 8531 at commit 5b2910b.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41818 has finished for PR 8531 at commit 5b2910b.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Given that we just had a build break in only one of the Hadoop branches due to a dep. change, I'm thinking that this script should generate four dependencies files, one for each Hadoop profile that we test on Jenkins.

[SparkQA]

Test build #44874 has started for PR 8531 at commit f4243b9.

[SparkQA]

Test build #44876 has finished for PR 8531 at commit 8b973bf.

• This patch fails executing the dev/run-tests script.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

Jenkins, test this please.

[pwendell]

Jenkins, test this please.

[SparkQA]

Test build #44886 has finished for PR 8531 at commit 2f4d3e5.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

Jenkins, retest this please.

[SparkQA]

Test build #44889 has finished for PR 8531 at commit 2f4d3e5.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44895 has finished for PR 8531 at commit ca686a3.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

We don't need runtime as well?

[JoshRosen]

Yep, I think you're right. According to the maven-dependency-plugin docs (https://maven.apache.org/plugins/maven-dependency-plugin/list-mojo.html#includeScope):

Scope to include. An Empty string indicates all scopes (default). The scopes being interpreted are the scopes as Maven sees them, not as specified in the pom. In summary:

• runtime scope gives runtime and compile dependencies,
• compile scope gives compile, provided, and system dependencies,
• test (default) scope gives all dependencies,
• provided scope just gives provided dependencies,
• system scope just gives system dependencies.

Based on this language, it seems like we want to be using runtime here so that we include both compile and runtime scope dependencies but do not exclude provided ones.

[SparkQA]

Test build #44907 has finished for PR 8531 at commit 37230f0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

Jenkins, test this please.

[SparkQA]

Test build #44950 has finished for PR 8531 at commit 1d91634.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Are you sure that we should sort the classpath? If two JARs provide the same classes then the ordering might matter and the relative classpath ordering might change as a consequence of adding a new dependency. Could also leave that up to followup, though, since this patch is massively useful as-is.

[JoshRosen]

I'm thinking of taking over this PR and am considering dropping this logic since it adds complexity and might not be a huge performance issue in practice. Let me know if you disagree.