Uh oh!
There was an error while loading. Please reload this page.
[SPARK-16729][SQL] Throw analysis exception for invalid date casts - #14358
[SPARK-16729][SQL] Throw analysis exception for invalid date casts#14358petermaxlee wants to merge 2 commits into
Conversation
SparkQA
commented
Jul 26, 2016
Test build #62857 has finished for PR 14358 at commit
|
petermaxlee
commented
Jul 26, 2016
@cloud-fan can you take a look? |
| // It is never possible to compare result when hive return with exception, | ||
| // so we can return null | ||
| // NULL is more reasonable here, since the query itself obeys the grammar. | ||
| case _ => _ => null |
There was a problem hiding this comment.
cc @yhuai@liancheng , do you remember why we have this behaviour at the beginning?
There was a problem hiding this comment.
Seems like a design decision made in the original PR. See here: https://github.com/apache/spark/pull/2344/files#diff-258b71121d8d168e4d53cb5b6dc53ffeR166
I don't think we've ever discussed this case explicitly. This change seems reasonable to me.
petermaxlee
commented
Jul 27, 2016
Is this good to merge? |
| // to ensure we test every possible cast situation here | ||
| atomicTypes.zip(atomicTypes).foreach { case (from, to) => | ||
| checkNullCast(from, to) | ||
| if (Cast.canCast(from, to)) { |
There was a problem hiding this comment.
why this check? doesn;t from always equal to to here?
There was a problem hiding this comment.
Not all atomicTypes can cast from each other? E.g. date.
There was a problem hiding this comment.
def canCast(from: DataType, to: DataType): Boolean = (from, to) match {
case (fromType, toType) if fromType == toType => true
......
There was a problem hiding this comment.
ah this is doing self casting - i read it wrong. let me remove it.
cloud-fan
commented
Jul 27, 2016
LGTM, pending jenkins. |
SparkQA
commented
Jul 27, 2016
Test build #62910 has finished for PR 14358 at commit
|
cloud-fan
commented
Jul 27, 2016
thanks, merging to master! |
rxin
commented
Jul 27, 2016
I would consider this a bug and put it in branch-2.0. I'm going to cherry pick. |
Spark currently throws exceptions for invalid casts for all other data types except date type. Somehow date type returns null. It should be consistent and throws analysis exception as well. Added a unit test case in CastSuite. Author: petermaxlee <petermaxlee@gmail.com> Closes#14358 from petermaxlee/SPARK-16729. (cherry picked from commit ef0ccbc) Signed-off-by: Reynold Xin <rxin@databricks.com>
What changes were proposed in this pull request?
Spark currently throws exceptions for invalid casts for all other data types except date type. Somehow date type returns null. It should be consistent and throws analysis exception as well.
How was this patch tested?
Added a unit test case in CastSuite.