Skip to content

[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators - #15704

Closed
dongjoon-hyun wants to merge 14 commits into
apache:masterfrom
dongjoon-hyun:SPARK-17732-2
Closed

[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators#15704
dongjoon-hyun wants to merge 14 commits into
apache:masterfrom
dongjoon-hyun:SPARK-17732-2

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Oct 31, 2016

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to support comparators, e.g. '<', '<=', '>', '>=', again in Apache Spark 2.0 for backward compatibility.

Spark 1.6

scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)")
res0: org.apache.spark.sql.DataFrame= [result: string]
scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")
res1: org.apache.spark.sql.DataFrame= [result: string]

Spark 2.0

scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country STRING, quarter STRING)")
res0: org.apache.spark.sql.DataFrame= []
scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '<' expecting {')', ','}(line 1, pos 42)

After this PR, it's supported.

How was this patch tested?

Pass the Jenkins test with a newly added testcase.

Loading
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@dongjoon-hyun@SparkQA@viirya@gatorsmile@hvanhovell