[SPARK-3654][SQL] Unifies SQL and HiveQL parsers by liancheng · Pull Request #2698 · apache/spark

liancheng · 2014-10-07T23:10:22Z

This PR is a follow up of #2590, and tries to introduce a top level SQL parser entry point for all SQL dialects supported by Spark SQL.

A top level parser SparkSQLParser is introduced to handle the syntaxes that all SQL dialects should recognize (e.g. CACHE TABLE, UNCACHE TABLE and SET, etc.). For all the syntaxes this parser doesn't recognize directly, it fallbacks to a specified function that tries to parse arbitrary input to a LogicalPlan. This function is typically another parser combinator like SqlParser. DDL syntaxes introduced in #2475 can be moved to here.

The ExtendedHiveQlParser now only handle Hive specific extensions.

Also took the chance to refactor/reformat SqlParser for better readability.

SparkQA · 2014-10-07T23:15:04Z

QA tests have started for PR 2698 at commit 53ae252.

This patch merges cleanly.

SparkQA · 2014-10-07T23:16:12Z

QA tests have finished for PR 2698 at commit 53ae252.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected case class Keyword(str: String)
- class SqlLexical(val keywords: Seq[String]) extends StdLexical
- case class FloatLit(chars: String) extends Token
- class SqlParser extends AbstractSparkSQLParser
- case class SetCommand(kv: Option[(String, Option[String])]) extends Command
- case class ShellCommand(cmd: String) extends Command
- case class SourceCommand(filePath: String) extends Command
- case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

AmplabJenkins · 2014-10-07T23:16:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21421/Test FAILed.

liancheng · 2014-10-07T23:16:49Z

Although looks trickier, Option[(String, Option[String])] captures the syntax and semantics of SET command better, and simplifies parsing logic.

liancheng · 2014-10-07T23:29:49Z

Scala style checking doesn't like the digit.+ DSL syntax in the parser, I'm falling back to rep1.

SparkQA · 2014-10-07T23:34:53Z

QA tests have started for PR 2698 at commit 9ebc290.

This patch merges cleanly.

liancheng · 2014-10-07T23:36:38Z

Debugging code, forgot to remove this...

SparkQA · 2014-10-07T23:44:34Z

QA tests have finished for PR 2698 at commit 9ebc290.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected case class Keyword(str: String)
- class SqlLexical(val keywords: Seq[String]) extends StdLexical
- case class FloatLit(chars: String) extends Token
- class SqlParser extends AbstractSparkSQLParser
- case class SetCommand(kv: Option[(String, Option[String])]) extends Command
- case class ShellCommand(cmd: String) extends Command
- case class SourceCommand(filePath: String) extends Command
- case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

AmplabJenkins · 2014-10-07T23:44:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21423/Test FAILed.

SparkQA · 2014-10-07T23:54:55Z

QA tests have started for PR 2698 at commit cc14609.

This patch merges cleanly.

SparkQA · 2014-10-08T00:05:45Z

QA tests have finished for PR 2698 at commit cc14609.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected case class Keyword(str: String)
- class SqlLexical(val keywords: Seq[String]) extends StdLexical
- case class FloatLit(chars: String) extends Token
- class SqlParser extends AbstractSparkSQLParser
- case class SetCommand(kv: Option[(String, Option[String])]) extends Command
- case class ShellCommand(cmd: String) extends Command
- case class SourceCommand(filePath: String) extends Command
- case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

AmplabJenkins · 2014-10-08T00:05:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21426/Test FAILed.

SparkQA · 2014-10-08T12:59:33Z

QA tests have started for PR 2698 at commit a4b0fc4.

This patch merges cleanly.

SparkQA · 2014-10-08T13:45:07Z

QA tests have finished for PR 2698 at commit a4b0fc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected case class Keyword(str: String)
- class SqlLexical(val keywords: Seq[String]) extends StdLexical
- case class FloatLit(chars: String) extends Token
- class SqlParser extends AbstractSparkSQLParser
- case class SetCommand(kv: Option[(String, Option[String])]) extends Command
- case class ShellCommand(cmd: String) extends Command
- case class SourceCommand(filePath: String) extends Command
- case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

AmplabJenkins · 2014-10-08T13:45:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21461/Test PASSed.

liancheng · 2014-10-09T04:22:34Z

@marmbrus Please help review. ADD JAR and ADD FILE are still Hive specific. I'd like to move them to the top level parser in another PR.

SparkQA · 2014-10-09T05:59:45Z

QA tests have started for PR 2698 at commit ba2c121.

This patch merges cleanly.

liancheng · 2014-10-09T06:13:00Z

This PR conflicts with #2678. It would be good to merge that one first, and then I'll rebase.

SparkQA · 2014-10-09T06:32:58Z

QA tests have finished for PR 2698 at commit ba2c121.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected case class Keyword(str: String)
- class SqlLexical(val keywords: Seq[String]) extends StdLexical
- case class FloatLit(chars: String) extends Token
- class SqlParser extends AbstractSparkSQLParser
- case class SetCommand(kv: Option[(String, Option[String])]) extends Command
- case class ShellCommand(cmd: String) extends Command
- case class SourceCommand(filePath: String) extends Command
- case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

AmplabJenkins · 2014-10-09T06:33:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21525/Test FAILed.

AmplabJenkins · 2014-10-09T20:32:18Z

Can one of the admins verify this patch?

SparkQA · 2014-10-10T00:19:42Z

QA tests have started for PR 2698 at commit ceada76.

This patch merges cleanly.

SparkQA · 2014-10-10T01:04:54Z

QA tests have finished for PR 2698 at commit ceada76.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected case class Keyword(str: String)
- class SqlLexical(val keywords: Seq[String]) extends StdLexical
- case class FloatLit(chars: String) extends Token
- class SqlParser extends AbstractSparkSQLParser
- case class SetCommand(kv: Option[(String, Option[String])]) extends Command
- case class ShellCommand(cmd: String) extends Command
- case class SourceCommand(filePath: String) extends Command
- case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(

AmplabJenkins · 2014-10-10T01:04:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21559/Test PASSed.

liancheng reviewed Oct 7, 2014
View reviewed changes

liancheng changed the title ~~[SPARK-3654][SQL][WIP] Unifies SQL and HiveQL parsers~~[SPARK-3654][SQL] Unifies SQL and HiveQL parsersOct 9, 2014

liancheng mentioned this pull request Oct 9, 2014
[SPARK-3645][SQL] Makes table caching eager by default and adds syntax for lazy caching #2513
Closed

liancheng added 7 commits October 10, 2014 08:04

Unifies SQL and HiveQL parsers
d15d54f

Replaces digit.+ to rep1(digit) (Scala style checking doesn't like it)
8bcace5

Removes debugging code
e86968e

Passes test suites
ce8860b

SET property value can be empty string
bb2ab12

Minor refactoring, removes optional trailing ";" in the parser
9738934

Minor styling fixes
ceada76

liancheng force-pushed the gen-sql-parser branch from ba2c121 to ceada76Compare October 10, 2014 00:15

asfgit closed this in edf02daOct 10, 2014

liancheng deleted the gen-sql-parser branch October 10, 2014 03:22

Uh oh!

Conversation

liancheng commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

AmplabJenkins commented Oct 7, 2014

Uh oh!

lianchengOct 7, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

lianchengOct 7, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

AmplabJenkins commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

liancheng commented Oct 9, 2014

Uh oh!

SparkQA commented Oct 9, 2014

Uh oh!

liancheng commented Oct 9, 2014

Uh oh!

SparkQA commented Oct 9, 2014

Uh oh!

AmplabJenkins commented Oct 9, 2014

Uh oh!

AmplabJenkins commented Oct 9, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

AmplabJenkins commented Oct 10, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants