[SPARK-5213] [SQL] Pluggable SQL Parser Support by scwf · Pull Request #5827 · apache/spark

scwf · 2015-05-01T05:03:25Z

based on #4015, we should not delete sqlParser from sqlcontext, that leads to mima failed. Users implement dialect to give a fallback for sqlParser and we should construct sqlParser in sqlcontext according to the dialect
protected[sql] val sqlParser = new SparkSQLParser(getSQLDialect().parse(_))

SparkQA · 2015-05-01T06:51:49Z

Test build #31518 has finished for PR 5827 at commit c19780b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class Dialect
- class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)
This patch does not change any dependencies.

chenghao-intel · 2015-05-01T15:03:19Z

Remove this comment? It's probably confused once this PR merged.

chenghao-intel · 2015-05-01T15:06:53Z

Thank you @scwf for the fixing. :)

scwf · 2015-05-01T15:27:33Z

@chenghao-intel this sqlparser actually will not be used for now, place here just to fix mima test

I think we'd better keep it, not just for the mima test, but also for the sub class of Dialect. e.g. we have to specify the SparkSQLParser for HiveQLDialect.

agree to keep it, and in dialect parser we should not use SparkSQLParser. Dialect give a fallback(string -> logicalplan) and we call it in sqlParser

scwf · 2015-05-01T15:45:55Z

Retest this please

SparkQA · 2015-05-01T17:27:55Z

Test build #31566 has finished for PR 5827 at commit 0878bd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class Dialect
- class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)

SparkQA · 2015-05-01T19:38:53Z

Test build #31573 timed out for PR 5827 at commit 81b9737 after a configured wait of 150m.

scwf · 2015-05-01T20:57:22Z

retest this please

SparkQA · 2015-05-01T22:47:22Z

Test build #31605 has finished for PR 5827 at commit 81b9737.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class Dialect
- class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)

scwf · 2015-05-02T00:20:22Z

@marmbrus any comment here?

This is a follow up of #5827 to remove the additional `SparkSQLParser` Author: Cheng Hao <hao.cheng@intel.com> Closes#5965 from chenghao-intel/remove_sparksqlparser and squashes the following commits: 509a233 [Cheng Hao] Remove the HiveQlQueryExecution a5f9e3b [Cheng Hao] Remove the duplicated SparkSQLParser (cherry picked from commit 074d75d) Signed-off-by: Michael Armbrust <michael@databricks.com>

This is a follow up of #5827 to remove the additional `SparkSQLParser` Author: Cheng Hao <hao.cheng@intel.com> Closes#5965 from chenghao-intel/remove_sparksqlparser and squashes the following commits: 509a233 [Cheng Hao] Remove the HiveQlQueryExecution a5f9e3b [Cheng Hao] Remove the duplicated SparkSQLParser

based on apache#4015, we should not delete `sqlParser` from sqlcontext, that leads to mima failed. Users implement dialect to give a fallback for `sqlParser` and we should construct `sqlParser` in sqlcontext according to the dialect `protected[sql] val sqlParser = new SparkSQLParser(getSQLDialect().parse(_))` Author: Cheng Hao <hao.cheng@intel.com> Author: scwf <wangfei1@huawei.com> Closesapache#5827 from scwf/sqlparser1 and squashes the following commits: 81b9737 [scwf] comment fix 0878bd1 [scwf] remove comments c19780b [scwf] fix mima tests c2895cf [scwf] Merge branch 'master' of https://github.com/apache/spark into sqlparser1 493775c [Cheng Hao] update the code as feedback 81a731f [Cheng Hao] remove the unecessary comment aab0b0b [Cheng Hao] polish the code a little bit 49b9d81 [Cheng Hao] shrink the comment for rebasing

This is a follow up of apache#5827 to remove the additional `SparkSQLParser` Author: Cheng Hao <hao.cheng@intel.com> Closesapache#5965 from chenghao-intel/remove_sparksqlparser and squashes the following commits: 509a233 [Cheng Hao] Remove the HiveQlQueryExecution a5f9e3b [Cheng Hao] Remove the duplicated SparkSQLParser

based on apache#4015, we should not delete `sqlParser` from sqlcontext, that leads to mima failed. Users implement dialect to give a fallback for `sqlParser` and we should construct `sqlParser` in sqlcontext according to the dialect `protected[sql] val sqlParser = new SparkSQLParser(getSQLDialect().parse(_))` Author: Cheng Hao <hao.cheng@intel.com> Author: scwf <wangfei1@huawei.com> Closesapache#5827 from scwf/sqlparser1 and squashes the following commits: 81b9737 [scwf] comment fix 0878bd1 [scwf] remove comments c19780b [scwf] fix mima tests c2895cf [scwf] Merge branch 'master' of https://github.com/apache/spark into sqlparser1 493775c [Cheng Hao] update the code as feedback 81a731f [Cheng Hao] remove the unecessary comment aab0b0b [Cheng Hao] polish the code a little bit 49b9d81 [Cheng Hao] shrink the comment for rebasing

This is a follow up of apache#5827 to remove the additional `SparkSQLParser` Author: Cheng Hao <hao.cheng@intel.com> Closesapache#5965 from chenghao-intel/remove_sparksqlparser and squashes the following commits: 509a233 [Cheng Hao] Remove the HiveQlQueryExecution a5f9e3b [Cheng Hao] Remove the duplicated SparkSQLParser

based on apache#4015, we should not delete `sqlParser` from sqlcontext, that leads to mima failed. Users implement dialect to give a fallback for `sqlParser` and we should construct `sqlParser` in sqlcontext according to the dialect `protected[sql] val sqlParser = new SparkSQLParser(getSQLDialect().parse(_))` Author: Cheng Hao <hao.cheng@intel.com> Author: scwf <wangfei1@huawei.com> Closesapache#5827 from scwf/sqlparser1 and squashes the following commits: 81b9737 [scwf] comment fix 0878bd1 [scwf] remove comments c19780b [scwf] fix mima tests c2895cf [scwf] Merge branch 'master' of https://github.com/apache/spark into sqlparser1 493775c [Cheng Hao] update the code as feedback 81a731f [Cheng Hao] remove the unecessary comment aab0b0b [Cheng Hao] polish the code a little bit 49b9d81 [Cheng Hao] shrink the comment for rebasing

This is a follow up of apache#5827 to remove the additional `SparkSQLParser` Author: Cheng Hao <hao.cheng@intel.com> Closesapache#5965 from chenghao-intel/remove_sparksqlparser and squashes the following commits: 509a233 [Cheng Hao] Remove the HiveQlQueryExecution a5f9e3b [Cheng Hao] Remove the duplicated SparkSQLParser

rxin · 2016-01-14T07:46:35Z

@scwf are you guys using this feature? I'm thinking about just removing it in Spark 2.0.

@chenghao-intel who wanted it in the first place no longer needs it.

scwf · 2016-01-14T15:10:20Z

@rxin, yes we used this and we implements a new sqlparser based on this interface to support ANSI tpcds sql.

rxin · 2016-01-14T18:07:47Z

What's different from the one in Spark master now? It would be great to contribute the parser changes back now we have a full fledged parser in Spark, and going towards more ANSI compatibility is definitely on the roadmap.

scwf · 2016-01-16T01:43:29Z

@rxin Our parser is a extended version of the SqlParser, the main difference is that we add the support for subquery(both correlated and uncorrelated ),exists, in and some minor improvement such as grouping, top, cube/rollup. It support the tpcds generated ANSI sql syntax without any change.

I noticed that there are some PRs for these features, i will take a look at that PRs when i have time and see what i can do.

rxin · 2016-01-16T02:18:24Z

FYI we are going to remove this pluggability. It is extra overhead to maintain, and actually encourages projects to not contribute their improvements upstream, which is bad.

scwf · 2016-01-16T03:30:52Z

Actually we were trying to contribute this improvements, unfortunately the community do not want them for maintain(or compatibility with hive ql) reason in the past:).

I am glad that spark sql use a single parser such that people can make contributions and make it more and more powerful.

rxin · 2016-01-18T05:22:44Z

Yup thanks. That's why we are only removing it now :)

chenghao-inteland others added 6 commits April 23, 2015 18:11

shrink the comment for rebasing
49b9d81

polish the code a little bit
aab0b0b

remove the unecessary comment
81a731f

update the code as feedback
493775c

Merge branch 'master' ofhttps://github.com/apache/sparkinto sqlparser1
c2895cf

fix mima tests
c19780b

chenghao-intel reviewed May 1, 2015
View reviewed changes

remove comments
0878bd1

scwf reviewed May 1, 2015
View reviewed changes

comment fix
81b9737

asfgit closed this in 5d6b90dMay 2, 2015

scwf deleted the sqlparser1 branch May 3, 2015 00:46

chenghao-intel mentioned this pull request May 7, 2015
[SPARK-5213] [SQL] Remove the duplicated SparkSQLParser #5965
Closed

ash211 mentioned this pull request Nov 18, 2016
Support for SQL dialects, in particular ANSI SQL palantir/spark#66
Closed

Uh oh!

Conversation

scwf commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

chenghao-intelMay 1, 2015

Choose a reason for hiding this comment

Uh oh!

scwfMay 1, 2015

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented May 1, 2015

Uh oh!

scwfMay 1, 2015

Choose a reason for hiding this comment

Uh oh!

chenghao-intelMay 1, 2015

Choose a reason for hiding this comment

Uh oh!

scwfMay 1, 2015

Choose a reason for hiding this comment

Uh oh!

scwf commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

scwf commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

scwf commented May 2, 2015

Uh oh!

rxin commented Jan 14, 2016

Uh oh!

scwf commented Jan 14, 2016

Uh oh!

rxin commented Jan 14, 2016

Uh oh!

scwf commented Jan 16, 2016

Uh oh!

rxin commented Jan 16, 2016

Uh oh!

scwf commented Jan 16, 2016

Uh oh!

rxin commented Jan 18, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants