Skip to content

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix) - #1620

Closed
liancheng wants to merge 2 commits into
apache:masterfrom
liancheng:jdbc-with-maven-fix
Closed

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix)#1620
liancheng wants to merge 2 commits into
apache:masterfrom
liancheng:jdbc-with-maven-fix

Conversation

@liancheng

Copy link
Copy Markdown
Contributor

JIRA issue: SPARK-2410

Another try for #1399 & #1600. Those two PR breaks Jenkins builds because we made a separate profile hive-thriftserver in sub-project assembly, but the hive-thriftserver module is defined outside the hive-thriftserver profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in hive-thriftserver, but tests fail because related .class files are not included in the assembly jar.

In the most recent commit, module hive-thriftserver is moved into its own profile to fix this problem. All previous commits are squashed for clarity.

Starts Hive Thrift server via spark-submit
Make HiveThriftServer2 play well with spark-submit
Starts spark-sql shell with spark-submit
Updated docs for Hive compatibility and Shark migration guide draft
Fixed minor issues in spark-sql and start-thriftserver.sh
Added missing license headers
Fixed more license header issues
Adapts test suites to spark-submit settings
Addressed all comments by @pwendell
Updated documents and build scripts for the newly added hive-thriftserver profile
Starts beeline with spark-submit
Fixed spark-submit application options handling logic
Any options in the application option list with the same option name
that SparkSubmitArguments recognizes (e.g., --help) are stolen by
SparkSubmit instead of passed to the application.
Fixed failed test suites
Disabled MIMA for hive-thriftserver
Reordered spark-submit options in spark-shell[.cmd]
All options behind primary resource are (and should be) recognized as
application options now.
Updated Spark SQL programming guide docs
Revert changes related to SPARK-2678, decided to move them to another PR
Uses random port for HiveThriftServer2 to avoid collision with parallel builds
@SparkQA

Copy link
Copy Markdown

QA tests have started for PR 1620. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17287/consoleFull

@SparkQA

Copy link
Copy Markdown

QA results for PR 1620:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17287/consoleFull

@marmbrus

Copy link
Copy Markdown
Contributor

Hopefully third times a charm :)

Merged!

@liancheng

Copy link
Copy Markdown
ContributorAuthor

Hmm, didn't find related code in master, and this PR is still open, seems it's not successfully merged yet?

@marmbrus

Copy link
Copy Markdown
Contributor

https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=a7a9d14479ea6421513a962ff0f45cb969368bab

GitHub is lagging... We filled a bug with infra.
On Jul 28, 2014 5:09 PM, "Cheng Lian" notifications@github.com wrote:

Hmm, didn't find related code in master, and this PR is still open, seems
it's not successfully merged yet?


Reply to this email directly or view it on GitHub
#1620 (comment).

@liancheng

Copy link
Copy Markdown
ContributorAuthor

Oh, really "lucky" these days :)

xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
… fix)
JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410)
Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar.
In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity.
Author: Cheng Lian <lian.cs.zju@gmail.com>
Closesapache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits:
629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile
ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server
@liancheng
liancheng deleted the jdbc-with-maven-fix branch September 24, 2014 00:09
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@liancheng@SparkQA@marmbrus