[SPARK-17473][SQL] fixing docker integration tests error due to different versions of jars.

[sureshthalamati]

What changes were proposed in this pull request?

Docker tests are using older version of jersey jars (1.19), which was used in older releases of spark. In 2.0 releases Spark was upgraded to use 2.x verison of Jersey. After upgrade to new versions, docker tests are failing with AbstractMethodError. Now that spark is upgraded to 2.x jersey version, using of shaded docker jars may not be required any more. Removed the exclusions/overrides of jersey related classes from pom file, and changed the docker-client to use regular jar instead of shaded one.

How was this patch tested?

Tested using existing docker-integration-tests

[sureshthalamati]

@JoshRosen

[SparkQA]

Test build #65461 has finished for PR 15114 at commit f159d82.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Let me give this a try locally and I'll report back whether it works for me. If so, I'll merge this.

[srowen]

I thought Jackson 1.x and 2.x were in different namespaces so I don't think they should be able to collide?

In any event, if it works, that's great, as it reduces the complexity here, but the root cause might be slightly different.

[lresende]

I verified this works on native docker in linux with :
build/mvn -Pdocker-integration-tests -Pscala-2.11 -pl :spark-docker-integration-tests_2.11 clean compile test

LGTM.

[JoshRosen]

I tried running this in SBT and ran into a bunch of spurious exceptions from logging code:

SLF4J: Failed toString() invocation on an object of type [com.spotify.docker.client.messages.ProgressMessage]
Reported exception:
java.lang.NoClassDefFoundError: com/google/common/base/MoreObjects
at com.spotify.docker.client.messages.ProgressMessage.toString(ProgressMessage.java:136)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:300)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:272)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:234)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:346)
at com.spotify.docker.client.LoggingPullHandler.progress(LoggingPullHandler.java:45)
at com.spotify.docker.client.ProgressStream.tail(ProgressStream.java:74)
at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:905)
at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:879)
at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:873)
at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:98)
at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
at org.apache.spark.SparkFunSuite.beforeAll(SparkFunSuite.scala:29)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:357)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:502)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

This didn't lead to test failures but made the test output a bit noisy while it was downloading images for the first time. This seems like it might be tricky to fix, though, so I'm going to merge this now to unblock developers who might rely on these tests. Thanks!