[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1 by scwf · Pull Request #2685 · apache/spark

scwf · 2014-10-07T01:49:43Z

In #2241 hive-thriftserver is not enabled. This patch enable hive-thriftserver to support hive-0.13.1 by using a shim layer refer to #2241.

1 A light shim layer(code in sql/hive-thriftserver/hive-version) for each different hive version to handle api compatibility

2 New pom profiles "hive-default" and "hive-versions"(copy from #2241) to activate different hive version

3 SBT cmd for different version as follows:
hive-0.12.0 --- sbt/sbt -Phive,hadoop-2.3 -Phive-0.12.0 assembly
hive-0.13.1 --- sbt/sbt -Phive,hadoop-2.3 -Phive-0.13.1 assembly

4 Since hive-thriftserver depend on hive subproject, this patch should be merged with #2241 to enable hive-0.13.1 for hive-thriftserver

AmplabJenkins · 2014-10-07T01:52:10Z

Can one of the admins verify this patch?

SparkQA · 2014-10-07T04:00:11Z

QA tests have started for PR 2685 at commit 3a08b14.

This patch merges cleanly.

SparkQA · 2014-10-07T04:01:13Z

QA tests have finished for PR 2685 at commit 3a08b14.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-09T20:32:24Z

Can one of the admins verify this patch?

scwf · 2014-10-13T07:39:27Z

@pwendell, i am resolving the conflicts, other TODO's here?

…riftserver

zhzhan · 2014-10-22T23:03:49Z

After hive-0.13.1 is committed, sql/hive-thirftserver can be put to top level instead of here.

you mean move to <modules> upper in this pom?

Yes. either move to modules, or move to hive profile, since it is supported in both versions.

…riftserver1

marmbrus · 2014-10-26T21:27:46Z

Now that Hive 13 is merged in it would be great to get this in ASAP. I looked over this and it seems pretty good. My only high level comment is maybe we should keep all the Hive Shim code in a single project instead of having version specific code in both hive and hive-thrift server. That would simplify the build and consolidate the places where we have these hacks. It woudl also allow us to avoid duplicating things like getCommandProcessor in both Shims. Thoughts?

@pwendell can you glance over the (limited) build changes.
@liancheng can you look this over as well?

pwendell · 2014-10-26T22:08:32Z

is this change ncessary? seems like it might be good to leave it how it is now

Yeah agreed. If Hive13 is the default we should test Hive13.

pwendell · 2014-10-26T22:12:03Z

Some minor comments on the build stuff. I think we're close, it's just small stuff.

The thirift server is not available in the default (hive13) profile yet which is breaking all SQL only PRs. This turns off these test until #2685 is merged. Author: Michael Armbrust <michael@databricks.com> Closes#2950 from marmbrus/fixTests and squashes the following commits: 1a6dfee [Michael Armbrust] [HOTFIX][SQL] Temporarily turn of hive-server tests.

scwf · 2014-10-27T03:43:19Z

@marmbrus, how about make a new sub project named hive-shim to keep all the Hive Shim code in it?

AmplabJenkins · 2014-10-31T05:52:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22591/
Test PASSed.

liancheng · 2014-10-31T05:57:10Z

Yay! It passed!

scwf · 2014-10-31T05:59:12Z

So will you re-publish the hive 0.13 jar @pwendell? or use 0.13.1a?

pwendell · 2014-10-31T06:01:48Z

it is not possible to mutate them after the fact. Let's stick with 0.13.1a - I will fully release it now. But don't remove the extra repository (we can remove it later) because it takes some time to propagate.

pwendell · 2014-10-31T06:04:00Z

I actually don't think this is compiling against Hive 0.12 right now... is it?

i think it's against 0.12 because BUILD_MVN_PROFILE_ARGS="$SBT_MAVEN_PROFILES_ARGS -Phive -Phive-0.12.0", right?

It should be, BUILD_MVN_PROFILE_ARGS is defined above with -Phive-0.12.0:
BUILD_MVN_PROFILE_ARGS="$SBT_MAVEN_PROFILES_ARGS -Phive -Phive-0.12.0"

zhzhan · 2014-10-31T06:15:01Z

whew, finally.

SparkQA · 2014-10-31T06:45:07Z

Test build #22602 has started for PR 2685 at commit f26f3be.

This patch merges cleanly.

liancheng · 2014-10-31T07:27:16Z

@zhzhan Had a glance at the Kryo issue you pointed out, it should be related to the POM inconsistency problem I mentioned above, but I'm not sure whether they are identical since this issue doesn't mention any specific Kryo version. I'll investigate this later. Thanks for pointing out this issue!

Although Jenkins finally nods, I'm still puzzled with the root cause of the original build failure. We only know that introducing Kryo 2.22 prevents un-shaded Objenesis classes from being included in the assembly jar and thus breaks the core tests, but how and why? Also, is it 100% safe to downgrade Kryo 2.22 to Kryo 2.21 for Hive 0.13.1?

SparkQA · 2014-10-31T08:29:51Z

Test build #22602 has finished for PR 2685 at commit f26f3be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-31T08:29:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22602/
Test PASSed.

marmbrus · 2014-10-31T18:27:30Z

Thanks guys for all your hard work on this! Merging to master.

coderfi · 2014-11-02T18:19:51Z

In /pom.xml

<properties>
<hive.version>0.13.1a</hive.version>
</properties>
...
<profile>
<id>hive-0.13.1</id>
...
<properties>
<hive.version>0.13.1</hive.version>
...
</properties>
</profile>

The 'hive-0.13.1' is overriding the hive.version back to the regular 'hive-0.13.1' version, instead of referencing 'hive-0.13.1a'.

It seems mvn -Phive would get me a build with hive-0.13a containing this pull's patches.

However, I was building with mvn -Phive -Phive-0.13.1, so I ran into the 'NoClassDefFoundError ... InstantiatorStrategy' issue.
Omitting the 'hive-0.13.1' profile got me past the issue (since the pom defaults effectively gives me hive 0.13(a) anyway, which is what I want).

liancheng · 2014-11-03T03:24:44Z

@coderfi This is a good catch, would you mind to file a JIRA ticket for this? A PR would be even better :)

scwf · 2014-11-03T04:05:33Z

Yes, here should be 0.13.1a.

pwendell · 2014-11-03T05:57:15Z

Yes - we just need to change that to 0.13.1a

On Sun, Nov 2, 2014 at 8:05 PM, wangfei notifications@github.com wrote:

Yes, here should be 0.13.1a.
—
Reply to this email directly or view it on GitHub
#2685 (comment).

scwf · 2014-11-03T06:17:19Z

And note just to change hive.version to 0.13.1a, hive.version.short should be 0.13.1.

coderfi · 2014-11-03T08:23:12Z

@liancheng PR #3072 created (all of one line! :) ).

instead of `hive.version=0.13.1`. e.g. mvn -Phive -Phive=0.13.1 Note: `hive.version=0.13.1a` is the default property value. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. References: PR #2685, which resolved a package incompatibility issue with Hive-0.13.1 by introducing a special version Hive-0.13.1a Author: fi <coderfi@gmail.com> Closes#3072 from coderfi/master and squashes the following commits: 7ca4b1e [fi] Fixes the `hive-0.13.1` maven profile referencing `hive.version=0.13.1` instead of the Spark compatible `hive.version=0.13.1a` Note: `hive.version=0.13.1a` is the default version. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. e.g. mvn -Phive -Phive=0.13.1 See PR #2685 (cherry picked from commit df607da) Signed-off-by: Michael Armbrust <michael@databricks.com>

instead of `hive.version=0.13.1`. e.g. mvn -Phive -Phive=0.13.1 Note: `hive.version=0.13.1a` is the default property value. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. References: PR #2685, which resolved a package incompatibility issue with Hive-0.13.1 by introducing a special version Hive-0.13.1a Author: fi <coderfi@gmail.com> Closes#3072 from coderfi/master and squashes the following commits: 7ca4b1e [fi] Fixes the `hive-0.13.1` maven profile referencing `hive.version=0.13.1` instead of the Spark compatible `hive.version=0.13.1a` Note: `hive.version=0.13.1a` is the default version. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. e.g. mvn -Phive -Phive=0.13.1 See PR #2685

scwf mentioned this pull request Oct 13, 2014
[SPARK-2706][SQL] Enable Spark to support Hive 0.13 #2241
Closed

scwf added 3 commits October 13, 2014 22:01

update and fix conflicts
ae47489

update pom according spark-2706
7c66b8e

Merge branch 'master' ofhttps://github.com/apache/sparkinto shim-th…
c6da3ce
…riftserver

scwf force-pushed the shim-thriftserver1 branch from 6cfcec8 to c6da3ceCompare October 14, 2014 16:48

scwf added 4 commits October 14, 2014 09:53

update run-tests to run hive-0.12.0 default now
dfd1c63

fixed when result filed is null
0bc53aa

enable thriftserver in profile hive-0.13.1
4b681f4

fix small bug
13afde0

zhzhan reviewed Oct 22, 2014
View reviewed changes

scwf added 2 commits October 22, 2014 19:25

revert pom changes
41f727b

Merge branch 'master' ofhttps://github.com/apache/sparkinto shim-th…
f48d3a5
…riftserver1

scwf force-pushed the shim-thriftserver1 branch from 9e5e2fc to f48d3a5Compare October 23, 2014 02:45

update and fix conflicts
f51ff4e

marmbrus mentioned this pull request Oct 26, 2014
[HOTFIX][SQL] Temporarily turn of hive-server tests. #2950
Closed

pwendell reviewed Oct 26, 2014
View reviewed changes

scwf added 2 commits October 26, 2014 20:07

move hive module to hive profile
3529e98

sql/hive included since examples depend on it
52674a4

pwendell reviewed Oct 31, 2014
View reviewed changes

remove clean to save time
f26f3be

liancheng mentioned this pull request Oct 31, 2014
[SPARK-3791][SQL] Provides Spark version and Hive version in HiveThriftServer2 #2843
Closed
1 task

asfgit closed this in 7c41d13Oct 31, 2014

scwf deleted the shim-thriftserver1 branch November 3, 2014 04:04

coderfi mentioned this pull request Nov 3, 2014
[SPARK-4211][Build] Fixes hive.version in Maven profile hive-0.13.1 #3072
Closed

liancheng mentioned this pull request Nov 14, 2014
[SPARK-4309][SPARK-4407][SQL] Date type support for Thrift server, and fixes for complex types #3178
Closed

adrian-wang mentioned this pull request Nov 25, 2014
[SPARK-4599] [Build] [SQL] add hive profile in root pom #3450
Closed

Uh oh!

Conversation

scwf commented Oct 7, 2014

Uh oh!

AmplabJenkins commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

AmplabJenkins commented Oct 9, 2014

Uh oh!

scwf commented Oct 13, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Oct 26, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwendell commented Oct 26, 2014

Uh oh!

scwf commented Oct 27, 2014

Uh oh!

AmplabJenkins commented Oct 31, 2014

Uh oh!

liancheng commented Oct 31, 2014

Uh oh!

scwf commented Oct 31, 2014

Uh oh!

pwendell commented Oct 31, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhzhan commented Oct 31, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

liancheng commented Oct 31, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

AmplabJenkins commented Oct 31, 2014

Uh oh!

marmbrus commented Oct 31, 2014

Uh oh!

coderfi commented Nov 2, 2014

Uh oh!

liancheng commented Nov 3, 2014

Uh oh!

scwf commented Nov 3, 2014

Uh oh!

pwendell commented Nov 3, 2014

Uh oh!

scwf commented Nov 3, 2014

Uh oh!

coderfi commented Nov 3, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants