Uh oh!
There was an error while loading. Please reload this page.
[SPARK-22016][SQL] Add HiveDialect for JDBC connection to Hive - #19238
[SPARK-22016][SQL] Add HiveDialect for JDBC connection to Hive#19238danielfx90 wants to merge 5 commits into
Conversation
AmplabJenkins
commented
Sep 14, 2017
Can one of the admins verify this patch? |
gatorsmile
commented
Sep 14, 2017
Why not directly connecting to Hive metastore? |
danielfx90
commented
Sep 15, 2017
@gatorsmile if Hive lies on the same infrastructure as the application, then the metastore should definitely solve the issue, but a connection over JDBC is needed when data comes from an external source which only exposes such a connection through its Hive server. We encountered this and ended up adding the HiveDialect to solve it. |
| assert(df3.collect() === Array(Row(21519, 1234))) | ||
| } | ||
| assert(df3.collect() === Array(Row(21519, 1234)) | ||
| ) |
There was a problem hiding this comment.
This ')' is wrong. Line 1105~1107 from the original have indentation issue.
There was a problem hiding this comment.
It must have changed when formatting the code using the IDE. Scalastyle checks passed though, but let me rollback that anyway.
There was a problem hiding this comment.
Ur, actually, I meant the original Spark code is also wrong in terms of indentation. You can fix the indentation of original line 1105~1107 here. :)
There was a problem hiding this comment.
@dongjoon-hyun You are right! I misread the parenthesis. I think now is correct. Thank you for the observation :)
gatorsmile
commented
Sep 18, 2017
I can see the value, but it does not perform well in most cases if we using JDBC connection. Instead of adding the extra dialect to upstream, could you please add Hive as a separate data source? Thanks! |
danielfx90
commented
Sep 18, 2017
Seems logical. Then, unless someone disagrees, feel free to close this PR and we will create a new spark package with this feature in a new repository. Thanks! |
paulstaab
commented
Jun 19, 2018
This merge request would partly solve https://issues.apache.org/jira/browse/SPARK-21063 |
Closesapache#13794Closesapache#18474Closesapache#18897Closesapache#18978Closesapache#19152Closesapache#19238Closesapache#19295Closesapache#19334Closesapache#19335Closesapache#19347Closesapache#19236Closesapache#19244Closesapache#19300Closesapache#19315Closesapache#19356Closesapache#15009Closesapache#18253 Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#19348 from HyukjinKwon/stale-prs.
What changes were proposed in this pull request?
Added a HiveDialect for JDBC connection to Hive.
It overrides two methods:
How was this patch tested?
It passes the added tests and it was used with a real Hive instance with real data.