[SPARK-16751] Upgrade derby to 10.12.1.1

[a-roberts]

What changes were proposed in this pull request?

Version of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1

The CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs

How was this patch tested?

Existing tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder.

I then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present.

I don't know if this would also remove it from the assembly jar in our 1.x branches.

[SparkQA]

Test build #62922 has finished for PR 14379 at commit 910520e.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[a-roberts]

Noticed the /home/jenkins/workspace/SparkPullRequestBuilder/dev/test-dependencies.sh problem, need to regen (I slipped up here when upgrading Hadoop last time too), on the case...

[shellberg]

Thanks Adam for putting together the PR - your more efficient at it than me ;-) - to propose upgrading this component and responding to my developer mailing list posting. I agree that this affects multiple prior Spark releases, and hoping this would make the cut for content towards a 1.6.x release at least too.

[a-roberts]

dev/test-dependencies.sh --replace-manifest then git status to see the changed files, updated the PR

[a-roberts]

Jenkins retest this please

[rxin]

Does this break the runtime for Spark SQL running locally with Hive support, when there is no external catalog setup?

[SparkQA]

Test build #62923 has finished for PR 14379 at commit 8016840.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #62924 has finished for PR 14379 at commit 8016840.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Yeah I don't think you can remove derby from packaging. If you don't have a metastore server, Spark (well, the Hive libraries) will create a local derby database and use it.

[a-roberts]

Interesting point, and you'd think that requires derby classes, out of curiosity what actually happens and what commands would we use to hit the problem?

I've noticed with bin/spark-shell we get a metastore_db folder and derby.log, without derby on our CP do we get NoClassDefFound exceptions? I can give this a try tomorrow if there's no quick answer here

If this is the case then we can simply remove the test I added and let's ship derby 10.12.1.1 not 10.11.1.1 instead

[vanzin]

I don't know what exact error you'd get, but you would almost certainly get some error. Removing the test scope (and updating the PR title) should be all that's needed here.

[SparkQA]

Test build #62940 has finished for PR 14379 at commit 3233b15.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #62943 has finished for PR 14379 at commit f3815cf.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Oh I see, so this is actually packaged, not strictly part of tests. Well we should do it anyway. Not sure if the risk actually surfaces in Spark but better to update and be safe.

[a-roberts]

One failure at https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62943/testReport/

which looks to be a timeout or flaky test and not caused by this PR, let's test once more to see if it consistent with this as the "age" on Jenkins doesn't help (usually gives us a history of this test failing)

[info] DirectKafkaStreamSuite:
[info] - basic stream receiving with multiple topics and smallest starting offset (6 seconds, 566 milliseconds)
[info] - pattern based subscription *** FAILED *** (21 seconds, 148 milliseconds)

[a-roberts]

Jenkins retest this please

[SparkQA]

Test build #62973 has finished for PR 14379 at commit f3815cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/2.0/1.6

[srowen]

Shoot, this made the lint check fail in 1.6 since there was a hadoop-1 dependency file to update there too. I'll hot-fix it forward, as that's an easy and trivial thing to correct.

[shellberg]

Thanks for merging into 1.6.x and the 2.X lines.