Uh oh!
There was an error while loading. Please reload this page.
[SPARK-20412] Throw ParseException from visitNonOptionalPartitionSpec instead of returning null values. - #17707
Closed
juliuszsompolski wants to merge 4 commits into
Closed
[SPARK-20412] Throw ParseException from visitNonOptionalPartitionSpec instead of returning null values.#17707juliuszsompolski wants to merge 4 commits into
juliuszsompolski wants to merge 4 commits into
Conversation
juliuszsompolski
commented
Apr 20, 2017
ContributorAuthor
cc @cloud-fan |
hvanhovell
commented
Apr 20, 2017
Contributor
ok to test |
hvanhovell
commented
Apr 20, 2017
Contributor
Could you add a unit test to the DDLCommandSuite? |
SparkQA
commented
Apr 20, 2017
Test build #75994 has finished for PR 17707 at commit
|
SparkQA
commented
Apr 20, 2017
Test build #75997 has finished for PR 17707 at commit
|
cloud-fan
commented
Apr 21, 2017
Contributor
looks like we need to fix a test |
| protected def visitNonOptionalPartitionSpec( | ||
| ctx: PartitionSpecContext): Map[String, String] = withOrigin(ctx) { | ||
| visitPartitionSpec(ctx).mapValues(_.orNull).map(identity) | ||
| ctx: PartitionSpecContext): Map[String, String] = withOrigin(ctx) { |
Member
There was a problem hiding this comment.
Nit: Indentation issues. Need to add extra two spaces
| visitPartitionSpec(ctx).mapValues(_.orNull).map(identity) | ||
| ctx: PartitionSpecContext): Map[String, String] = withOrigin(ctx) { | ||
| visitPartitionSpec(ctx).map { | ||
| case (key, None) => throw new ParseException(s"Found empty key '$key'.", ctx) |
Member
There was a problem hiding this comment.
How about Found an empty partition key '$key'?
SparkQA
commented
Apr 21, 2017
Test build #76029 has finished for PR 17707 at commit
|
| tableIdent, | ||
| None, | ||
| Some(Map("columns" -> "foo,bar", "field.delim" -> ",")), | ||
| Some(Map("test" -> null, "dt" -> "2008-08-08", "country" -> "us"))) |
ContributorAuthor
There was a problem hiding this comment.
BTW: In AlterTableSerDePropertiesCommand that null would have caused a NullPointerExeception later, so not having this as null, but checked as ParseException is also a valid change here.
cloud-fan
commented
Apr 21, 2017
Contributor
thanks, merging to master/2.2! |
asfgit pushed a commit
that referenced
this pull request
Apr 21, 2017
… instead of returning null values. ## What changes were proposed in this pull request? If a partitionSpec is supposed to not contain optional values, a ParseException should be thrown, and not nulls returned. The nulls can later cause NullPointerExceptions in places not expecting them. ## How was this patch tested? A query like "SHOW PARTITIONS tbl PARTITION(col1='val1', col2)" used to throw a NullPointerException. Now it throws a ParseException. Author: Juliusz Sompolski <julek@databricks.com> Closes#17707 from juliuszsompolski/SPARK-20412. (cherry picked from commit c9e6035) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
peter-toth pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
… instead of returning null values. ## What changes were proposed in this pull request? If a partitionSpec is supposed to not contain optional values, a ParseException should be thrown, and not nulls returned. The nulls can later cause NullPointerExceptions in places not expecting them. ## How was this patch tested? A query like "SHOW PARTITIONS tbl PARTITION(col1='val1', col2)" used to throw a NullPointerException. Now it throws a ParseException. Author: Juliusz Sompolski <julek@databricks.com> Closesapache#17707 from juliuszsompolski/SPARK-20412.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
If a partitionSpec is supposed to not contain optional values, a ParseException should be thrown, and not nulls returned.
The nulls can later cause NullPointerExceptions in places not expecting them.
How was this patch tested?
A query like "SHOW PARTITIONS tbl PARTITION(col1='val1', col2)" used to throw a NullPointerException.
Now it throws a ParseException.