[SPARK-3124] Fix the jar version conflict in uber-jar by chenghao-intel · Pull Request #2035 · apache/spark

chenghao-intel · 2014-08-19T15:37:38Z

Both netty-3.2.2.Final.jar and netty-3.6.6.Final.jar are flatten into the assembly package, however, the class(NioWorker) signature difference leads to the failure in launching sparksql CLI/ThriftServer.

And also a tiny bug found in CLISuite & HiveThriftServer2Suite after fixing the duplicated netty jar.

SparkQA · 2014-08-19T15:40:19Z

QA tests have started for PR 2035 at commit fe62aa9.

This patch merges cleanly.

SparkQA · 2014-08-19T15:45:20Z

QA tests have started for PR 2035 at commit 7d9e366.

This patch merges cleanly.

witgo · 2014-08-19T15:53:18Z

There should not be modified.
We should modify the file sql/hive-thriftserver/pom.xml

<dependency> <groupId>org.spark-project.hive</groupId> <artifactId>hive-cli</artifactId> <version>${hive.version}</version> <exclusions> <exclusion> <groupId>org.jboss.netty</groupId> <artifactId>netty</artifactId> </exclusion> </exclusions> </dependency>

SparkQA · 2014-08-19T16:49:31Z

QA tests have finished for PR 2035 at commit fe62aa9.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-08-19T16:57:19Z

QA tests have finished for PR 2035 at commit 7d9e366.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-08-20T07:29:27Z

QA tests have started for PR 2035 at commit 182b52b.

This patch merges cleanly.

SparkQA · 2014-08-20T08:25:04Z

QA tests have finished for PR 2035 at commit 182b52b.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2014-08-20T08:30:07Z

test this please.

SparkQA · 2014-08-20T08:35:26Z

QA tests have started for PR 2035 at commit 81fc0d7.

This patch merges cleanly.

SparkQA · 2014-08-20T08:35:33Z

QA tests have finished for PR 2035 at commit 81fc0d7.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-08-20T18:16:31Z

test this please

pwendell · 2014-08-20T18:18:55Z

I commented on the JIRA - but this is likely better solved by a dependency in the hive build, not changing around core dependencies.

SparkQA · 2014-08-20T18:20:32Z

QA tests have started for PR 2035 at commit 81fc0d7.

This patch merges cleanly.

SparkQA · 2014-08-20T18:20:38Z

QA tests have finished for PR 2035 at commit 81fc0d7.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2014-08-21T01:09:55Z

Make sure the hive test will be triggered.

chenghao-intel · 2014-08-21T01:11:51Z

@pwendell I've updated the jira. You are right, it works well with hadoop 2.3, but failed under hadoop-2.0.0-mr1-cdh4.3.0, which I am working with.

SparkQA · 2014-08-21T01:15:36Z

QA tests have started for PR 2035 at commit ade1ae0.

This patch merges cleanly.

chenghao-intel · 2014-08-21T01:21:51Z

I was trying to update the pom.xml under sql/, but still failed in evicting the netty.3.2.2.Final.jar, and at last, I printed the it's dependency tree, and found the spark-core indirectly depends on netty.3.2.2.Final.jar via hadoop-client, curator, zookeeper etc., hence I have to updated the pom.xml under root and core. Not sure if you have better idea to solve this.

SparkQA · 2014-08-21T02:33:19Z

QA tests have finished for PR 2035 at commit ade1ae0.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2014-08-21T03:42:25Z

It passed the unit test in my local.
Jenkins, test this please

chenghao-intel · 2014-08-21T05:04:34Z

test this please

SparkQA · 2014-08-21T05:10:34Z

QA tests have started for PR 2035 at commit ade1ae0.

This patch merges cleanly.

witgo · 2014-08-21T06:19:15Z

I think we need to modify this file: sql/hive-thriftserver/pom.xml

    <dependency>
      <groupId>org.spark-project.hive</groupId>
      <artifactId>hive-cli</artifactId>
      <version>${hive.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.jboss.netty</groupId>
          <artifactId>netty</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

SparkQA · 2014-08-21T06:33:20Z

QA tests have finished for PR 2035 at commit ade1ae0.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2014-08-21T06:49:16Z

I pulled the latest master and only update the sql/hive-thriftserver/pom.xml as you suggested, however, both netty-3.2.2.Final.jar & netty-3.6.6.Final.jar exist under the folder lib_managed/bundled by running sbt/sbt -Phive -Phive-thriftserver -Dhadoop.version=2.0.0-mr1-cdh4.3.0 clean compile.

That's why I changed the pom.xml files under root and core. Not sure if you got time to reproduce that in your local?

chenghao-intel · 2014-08-21T06:53:38Z

Sorry for the confusing, I've removed the SQL from the PR title.

witgo · 2014-08-21T06:54:36Z

./make-distribution.sh -Pyarn -Phadoop-2.3 -Phive-thriftserver -Phive -Dhadoop.version=2.3.0.

./bin/spark-sql --hiveconf hive.root.logger=INFO,console seems to be no problem,under the folder dist

chenghao-intel · 2014-08-21T06:58:24Z

Exactly, I did the same test under hadoop-2.3, it plays well, but not for hadoop.version=2.0.0-mr1-cdh4.3.0. Probably the maven exclusion rule are not well configured for the earlier released hadoop.

marmbrus · 2014-09-03T03:43:42Z

@chenghao-intel, what is the status of this? Can you update this PR?

SparkQA · 2014-09-03T14:04:45Z

QA tests have started for PR 2035 at commit f96ec21.

This patch merges cleanly.

chenghao-intel · 2014-09-03T14:16:24Z

Minor change for triggering the HQL testing.

chenghao-intel · 2014-09-03T14:28:40Z

@marmbrus this PR will fix method signature exception for the earlier released hadoop version in the uber-jar (like 2.0.0-mr1-cdh4.3.0), and it's harmless for current trunk(which is based on hadoop 2.3.0 for unit test). I created this PR just in case someone making uber-jar with different hadoop version other than default, which may causes runtime exception, like what I did.

@pwendell @witgo , any ideas?

SparkQA · 2014-09-03T15:50:44Z

QA tests have finished for PR 2035 at commit f96ec21.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected trait YarnAllocateResponse

chenghao-intel · 2014-09-04T00:38:02Z

retest this please.

SparkQA · 2014-09-04T00:44:32Z

QA tests have started for PR 2035 at commit f96ec21.

This patch merges cleanly.

SparkQA · 2014-09-04T02:25:01Z

QA tests have finished for PR 2035 at commit f96ec21.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2014-09-04T08:55:50Z

test this please

SparkQA · 2014-09-04T08:59:24Z

QA tests have started for PR 2035 at commit f96ec21.

This patch merges cleanly.

witgo · 2014-09-04T09:52:35Z

LGTM

SparkQA · 2014-09-04T10:55:44Z

QA tests have finished for PR 2035 at commit f96ec21.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2014-09-05T00:58:32Z

thank you @witgo. @pwendell I think this is ready to be merged.

SparkQA · 2014-09-06T00:05:22Z

QA tests have started for PR 2035 at commit f96ec21.

This patch merges cleanly.

SparkQA · 2014-09-06T02:05:23Z

Tests timed out after a configured wait of 120m.

SparkQA · 2014-09-15T02:10:34Z

QA tests have started for PR 2035 at commit f96ec21.

This patch merges cleanly.

SparkQA · 2014-09-15T03:37:39Z

QA tests have finished for PR 2035 at commit f96ec21.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

kayousterhout · 2014-09-18T05:52:22Z

When building against Hadoop-2.0.2, I found that only this dependency exclusion (not the additional one you included below) was necessary to get past the error you mentioned.

chenghao-intel · 2014-10-30T02:20:17Z

thanks, closing this PR.

witgo reviewed Aug 19, 2014
View reviewed changes

chenghao-intel force-pushed the assembly_jar_conflict branch from 7d9e366 to 182b52b Compare August 20, 2014 07:23

chenghao-intel mentioned this pull request Aug 20, 2014

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

Closed

chenghao-intel force-pushed the assembly_jar_conflict branch from 81fc0d7 to ade1ae0 Compare August 21, 2014 01:08

chenghao-intel reviewed Aug 21, 2014
View reviewed changes

chenghao-intel changed the title ~~[SPARK-3124] [SQL] Fix the assembly jar conflict~~ [SPARK-3124] Fix the jar version conflict in uber-jar Aug 21, 2014

chenghao-intel added 2 commits September 3, 2014 21:08

Remove the netty.3.2.2.Final dependency

e397129

trigger the unit test for HiveContext

f96ec21

chenghao-intel force-pushed the assembly_jar_conflict branch from ade1ae0 to f96ec21 Compare September 3, 2014 13:58

chenghao-intel reviewed Sep 3, 2014
View reviewed changes

kayousterhout reviewed Sep 18, 2014
View reviewed changes

chenghao-intel closed this Oct 30, 2014

chenghao-intel deleted the assembly_jar_conflict branch December 10, 2014 02:42

Conversation

chenghao-intel commented Aug 19, 2014

Uh oh!

SparkQA commented Aug 19, 2014

Uh oh!

SparkQA commented Aug 19, 2014

Uh oh!

witgo Aug 19, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 19, 2014

Uh oh!

SparkQA commented Aug 19, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

chenghao-intel commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

marmbrus commented Aug 20, 2014

Uh oh!

pwendell commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

chenghao-intel Aug 21, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

SparkQA commented Aug 21, 2014

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

SparkQA commented Aug 21, 2014

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

SparkQA commented Aug 21, 2014

Uh oh!

witgo commented Aug 21, 2014

Uh oh!

SparkQA commented Aug 21, 2014

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

witgo commented Aug 21, 2014

Uh oh!

chenghao-intel commented Aug 21, 2014

Uh oh!

marmbrus commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

chenghao-intel Sep 3, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

chenghao-intel commented Sep 4, 2014

Uh oh!

SparkQA commented Sep 4, 2014

Uh oh!

SparkQA commented Sep 4, 2014

Uh oh!

chenghao-intel commented Sep 4, 2014

Uh oh!