[SPARK-9818] Re-enable Docker tests for JDBC data source by JoshRosen · Pull Request #9503 · apache/spark

JoshRosen · 2015-11-05T21:17:18Z

This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the docker-client library. This patch should avoid those problems by using a version of docker-client which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR.

In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in boot2docker.

Closes#8101.

JoshRosen · 2015-11-05T21:17:25Z

/cc @marmbrus for review.

JoshRosen · 2015-11-05T21:19:04Z

Although the tests should automatically pull any missing Docker images, I went ahead and manually pulled them on all of the AMP Jenkins workers. This uncovered an issue where two of the workers' Docker filesystems had entered a safe mode that prevented image pulling, so I logged in and fixed that.

marmbrus · 2015-11-05T21:26:10Z

I always forget your can do this...

marmbrus · 2015-11-05T21:38:11Z

For posterity, is there any easy way to do the parallel pull if we upgrade one of the images or add a new one?

JoshRosen · 2015-11-05T21:39:42Z

Here's the command that I used:

pssh -O StrictHostKeyChecking=no -h /root/jenkins_workers.txt -i -x '-t -t' "sudo docker pull mysql:5.7.9"

marmbrus · 2015-11-05T21:45:07Z

Tested this locally and other than needing to set DOCKER_IP everything worked great. Talked ot @JoshRosen offline and he's going to try and automate this.

SparkQA · 2015-11-05T21:57:11Z

Test build #1990 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-05T21:58:05Z

Test build #45145 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-05T22:01:15Z

Test build #1992 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-05T22:04:09Z

Test build #1991 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

JoshRosen · 2015-11-06T00:24:33Z

Woah, this is an ancient version of docker-client. I'll try using a newer release to see if that magically fixes the ASM issues.

SparkQA · 2015-11-06T00:39:13Z

Test build #45172 has finished for PR 9503 at commit 95f00e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-06T03:12:23Z

Test build #45176 has finished for PR 9503 at commit fcf5dc4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

…sues.

SparkQA · 2015-11-06T09:23:10Z

Test build #45214 has finished for PR 9503 at commit 2273f87.

This patch fails to build.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-11-06T09:35:30Z

Test build #45215 has finished for PR 9503 at commit 65315c4.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

JoshRosen · 2015-11-10T06:32:51Z

Something really weird is going on here.

Following the advice at http://stackoverflow.com/a/21876380/590203, I added the following line to DockerJDBCIntegrationSuite in order to figure out where BasicLineFormatter was being loaded from:

println(this.getClass.getClassLoader.getResource("org/apache/http/message/BasicLineFormatter.class"))

When I run this locally with

 sbt/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive

I get

jar:file:/Users/joshrosen/Documents/spark/lib_managed/jars/httpcore-4.2.4.jar!/org/apache/http/message/BasicLineFormatter.class

This doesn't make sense, though: show test:dependencyClasspath does not include that JAR, so I'm not quite sure how it's winding up on the classpath. If I toss in a

 this.getClass.getClassLoader.asInstanceOf[URLClassLoader].getURLs.foreach(println)

I still don't see where that JAR is coming from.

JoshRosen · 2015-11-10T07:10:29Z

In fact, httpcore-4.2.4 doesn't even show up in the -classpath argument passed to ForkMain.

I used some of the handy classpath-debugging tools in JHades to dump more information and it looks like httpcore-4.2.4 is included in the Tachyon JARs' MANIFEST.MF files. Here's the manfest from tachyon-client:

Manifest-Version: 1.0
Built-By: DreamSweet
Build-Jdk: 1.7.0_75
Class-Path: guava-14.0.1.jar commons-lang-2.4.jar commons-io-2.4.jar c
ommons-lang3-3.0.jar curator-framework-2.1.0-incubating.jar zookeeper
-3.4.5.jar slf4j-log4j12-1.7.2.jar log4j-1.2.15.jar mail-1.4.jar acti
vation-1.1.jar jline-0.9.94.jar netty-3.2.2.Final.jar curator-client-
2.1.0-incubating.jar curator-recipes-2.1.0-incubating.jar slf4j-api-1
.7.2.jar hadoop-client-1.0.4.jar hadoop-core-1.0.4.jar xmlenc-0.52.ja
r commons-codec-1.4.jar commons-math-2.1.jar commons-configuration-1.
6.jar commons-collections-3.2.1.jar commons-digester-1.8.jar commons-
beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-net-1.4.
1.jar commons-el-1.0.jar hsqldb-1.8.0.10.jar oro-2.0.8.jar jackson-ma
pper-asl-1.0.1.jar jackson-core-asl-1.0.1.jar tachyon-common-0.8.1.ja
r netty-all-4.0.28.Final.jar libthrift-0.9.2.jar httpclient-4.2.5.jar
httpcore-4.2.4.jar tachyon-client-unshaded-0.8.1.jar tachyon-underfs
-hdfs-0.8.1.jar tachyon-underfs-glusterfs-0.8.1.jar glusterfs-hadoop-
2.3.13.jar hadoop-yarn-server-nodemanager-2.3.0.jar hadoop-yarn-commo
n-2.3.0.jar hadoop-yarn-api-2.3.0.jar jaxb-api-2.2.2.jar stax-api-1.0
-2.jar jettison-1.1.jar servlet-api-2.5.jar jersey-core-1.9.jar jerse
y-client-1.9.jar jetty-util-6.1.26.jar hadoop-annotations-2.3.0.jar g
uice-servlet-3.0.jar protobuf-java-2.5.0.jar guice-3.0.jar javax.inje
ct-1.jar aopalliance-1.0.jar jersey-json-1.9.jar jaxb-impl-2.2.3-1.ja
r jackson-jaxrs-1.8.3.jar jackson-xc-1.8.3.jar jersey-guice-1.9.jar h
adoop-yarn-server-common-2.3.0.jar hadoop-common-2.3.0.jar commons-cl
i-1.2.jar commons-math3-3.1.1.jar jetty-6.1.26.jar jersey-server-1.9.
jar asm-3.1.jar jasper-compiler-5.5.23.jar jasper-runtime-5.5.23.jar jsp-api-2.1.jar avro-1.7.4.jar paranamer-2.3.jar snappy-java-1.0.4.1.
jar hadoop-auth-2.3.0.jar jsch-0.1.42.jar jsr305-1.3.9.jar commons-co
mpress-1.4.1.jar xz-1.0.jar tachyon-underfs-s3-0.8.1.jar jets3t-0.8.1
.jar commons-logging-1.1.1.jar commons-httpclient-3.1.jar java-xmlbui
lder-0.4.jar tachyon-underfs-local-0.8.1.jar
Created-By: Apache Maven 3.3.3
Archiver-Version: Plexus Archiver

According to the Java 7 JAR specification (emphasis mine):

Class-Path: The value of this attribute specifies the relative URLs of the extensions or libraries that this application or extension needs. URLs are separated by one or more spaces. The application or extension class loader uses the value of this attribute to construct its internal search path.

Assuming that "relative" means "relative to the JAR containing the manifest", I think that what's happening is that this is causing the httpcore-4.2.4 JAR from lib_managed to be added to the classpath, which explains why testing this patch after rebasing on top of #9575 (which minimizes our use of lib_managed/jars to only include the Datanucleus JARs) seemed to fix things.

What a giant mess. It looks like the <addClassPath> manifest configuration ended up being added to Tachyon's root POM as part of Alluxio/alluxio#939.

pwendell · 2015-11-10T07:22:14Z

I only reviewed the build changes, but they look good to me.

JoshRosen · 2015-11-10T07:31:14Z

I found this Manifest class-path issue to be extremely puzzling, but it looks like other people have hit it as well: https://dzone.com/articles/jar-manifest-class-path-is-not-for-java-applicatio

Quote:

A JAR with a Class-Path manifest header will make those classpath entries available to the Java classloader regardless of whether that JAR is executed with java -jar ... or is simply placed on the classpath of a larger Java application.

This puzzling behavior is described a bit more explicitly in Java 8's JAR specification.

SparkQA · 2015-11-10T07:39:10Z

Test build #45495 has finished for PR 9503 at commit 9011bc5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-11-10T07:43:44Z

Created https://tachyon.atlassian.net/browse/TACHYON-1254 for this; this will be worked around in Spark via my lib_managed patch, but I still feel that it should be fixed in Tachyon as well so that it doesn't bite anyone in the future.

JoshRosen · 2015-11-10T08:10:30Z

And just to be 100%-sure that it was the Tachyon manifests's fault, I used the following command to delete them and verified that doing so fixed the tests:

cd $SPARK_HOME
ls -d -1 lib_managed/jars/* | grep tachyon | xargs -n1 -I{} zip {} --delete META-INF/MANIFEST.MF

JoshRosen · 2015-11-10T18:21:37Z

Jenkins, retest this please.

SparkQA · 2015-11-10T20:23:33Z

Test build #45536 has finished for PR 9503 at commit 9011bc5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-11-10T20:24:26Z

Looks like a flaky streaming test?

Jenkins, retest this please.

SparkQA · 2015-11-10T23:14:02Z

Test build #2030 has finished for PR 9503 at commit 9011bc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-11-10T23:58:21Z

I'm merging this. Thanks.

This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the `docker-client` library. This patch should avoid those problems by using a version of `docker-client` which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR. In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in `boot2docker`. Closes#8101. Author: Josh Rosen <joshrosen@databricks.com> Author: Yijie Shen <henry.yijieshen@gmail.com> Closes#9503 from JoshRosen/docker-jdbc-tests. (cherry picked from commit 1dde39d) Signed-off-by: Reynold Xin <rxin@databricks.com>

tedyu · 2015-11-13T10:44:16Z

From https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/4095/consoleFull :

MySQLIntegrationSuite:
*** RUN ABORTED ***
java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
at org.glassfish.jersey.apache.connector.ApacheConnector.<init>(ApacheConnector.java:240)
at org.glassfish.jersey.apache.connector.ApacheConnectorProvider.getConnector(ApacheConnectorProvider.java:115)
at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:418)
at org.glassfish.jersey.client.ClientConfig$State.access$000(ClientConfig.java:88)
at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:120)
at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:117)
at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:340)
at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:726)
at org.glassfish.jersey.client.ClientRequest.getConfiguration(ClientRequest.java:285)
at org.glassfish.jersey.client.JerseyInvocation.validateHttpMethodAndEntity(JerseyInvocation.java:126)

markhamstra · 2015-11-13T19:04:25Z

Similar failure with both PostgresIntegrationSuite and MySQLIntegrationSuite from dev/run-tests on OSX 10.11.1:

[info] Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.jdbc.PostgresIntegrationSuite *** ABORTED *** (3 seconds, 40 milliseconds)
[info] com.spotify.docker.client.DockerException: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:2375 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1, localhost/fe80:0:0:0:0:0:0:1%1] failed: Connection refused
[info] at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1141)
[info] at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1082)
[info] at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:281)
[info] at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:76)
[info] at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
...

JoshRosen · 2015-11-13T19:27:34Z

Maybe the docker daemon didn't get started / restarted on a subset of the machines?

tedyu · 2015-11-14T00:08:47Z

Should MySQLIntegrationSuite be disabled since the test failure is reproducible ?

JoshRosen · 2015-11-14T02:09:19Z

If you don't have Docker set up locally, you can use the DockerTest test tag to exclude those tests.

lvc · 2016-10-03T14:04:45Z

@JoshRosen

The backward compatibility reports for Jersey* libraries are now available in the API Tracker project:

https://abi-laboratory.pro/java/tracker/timeline/jersey-core/
https://abi-laboratory.pro/java/tracker/timeline/jersey-server/

The JAPICC tool detected some servlet related changes in jersey-core 1.10:
https://abi-laboratory.pro/java/tracker/compat_report/jersey-server/1.9.1/1.10/2ae0a/bin_compat_report.html

nonsleepr · 2018-05-22T21:18:41Z

Adding following for better visibility.

The problem while using com.whisk:docker-testkit-impl-spotify along with Spark:

com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2657)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2588)
at com.spotify.docker.client.DefaultDockerClient.info(DefaultDockerClient.java:522)
... 40 elided
Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2586)
... 41 more
Caused by: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:202)
at org.glassfish.jersey.client.ClientRuntime.access$400(ClientRuntime.java:79)
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:182)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:74)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:59)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:43)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:287)
at jnr.unixsocket.Native.<clinit>(Native.java:76)
at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:68)
at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:49)
at com.spotify.docker.client.ApacheUnixSocket.<init>(ApacheUnixSocket.java:59)
at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:67)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
... 12 more

The solution is to use shaded package:

libraryDependencies +="com.spotify"%"docker-client"%"8.11.3" classifier "shaded"

yjshenand others added 8 commits October 18, 2015 18:00

Import yjshen's PR.
4b82a2c

Use newer shaded docker-client artifact
eddcc47

Use map for setting environment variables.
8bb62ea

Ensure that Docker container is cleaned up after error.
23443a5

Remove a bunch of layers of indirection / abstraction.
e6c7709

Fix networking for boot2docker.
d06847b

Log warning in case container cleanup fails
93bcf45

Rename class and freeze image versions
bf98ae0

JoshRosen mentioned this pull request Nov 5, 2015
[SPARK-9818][SQL][WIP]Revert SPARK-6136 to enable jdbc tests using docker #8101
Closed

marmbrus reviewed Nov 5, 2015
View reviewed changes

Automatically get docker IP from docker-machine and boot2docker.
1e389e2

Print nicer message if Docker connection fails.
d9c37df

Add test tag for excluding Docker tests in SBT.
95f00e1

JoshRosen reviewed Nov 6, 2015
View reviewed changes

Upgrade to non-ancient version of docker-client.
fcf5dc4

Upgrade Jersey to avoid ASM incompatibilities / classpath ordering is…
2273f87
…sues.

Merge remote-tracking branch 'origin/master' into docker-jdbc-tests
65315c4

Merge remote-tracking branch 'origin/master' into docker-jdbc-tests
9011bc5

JoshRosen mentioned this pull request Nov 10, 2015
Migrate Spark SQL JDBC data source integration tests here databricks/spark-integration-tests#6
Closed

asfgit closed this in 1dde39dNov 11, 2015

This was referenced Nov 11, 2015
[SPARK-10186][SQL][WIP] Array types using JDBCRDD and postgres #9137
Closed
[SPARK-10170] [SQL] Add DB2 JDBC dialect support. #8393
Closed
[SPARK-10655][SQL] Adding additional data type mappings to jdbc DB2dialect. #9162
Closed

JoshRosen deleted the docker-jdbc-tests branch April 20, 2016 23:58

Uh oh!

Conversation

JoshRosen commented Nov 5, 2015

Uh oh!

JoshRosen commented Nov 5, 2015

Uh oh!

JoshRosen commented Nov 5, 2015

Uh oh!

marmbrusNov 5, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Nov 5, 2015

Uh oh!

JoshRosen commented Nov 5, 2015

Uh oh!

marmbrus commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

JoshRosenNov 6, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

pwendell commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

rxin commented Nov 10, 2015

Uh oh!

tedyu commented Nov 13, 2015

Uh oh!

markhamstra commented Nov 13, 2015

Uh oh!

JoshRosen commented Nov 13, 2015

Uh oh!

tedyu commented Nov 14, 2015

Uh oh!

JoshRosen commented Nov 14, 2015

Uh oh!

lvc commented Oct 3, 2016

Uh oh!

nonsleepr commented May 22, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development