Uh oh!
There was an error while loading. Please reload this page.
[SPARK-12841][SQL] fix cast in filter - #10781
Conversation
cloud-fan
commented
Jan 16, 2016
cloud-fan
commented
Jan 16, 2016
retest this please |
SparkQA
commented
Jan 16, 2016
Test build #49506 has finished for PR 10781 at commit
|
There was a problem hiding this comment.
We use transformUp at here to make this logic equivalent with the original logic that we have in the cast, right (since we want to fix it in 1.6)? (in future, will it be possible to just use cast Cast(ne: NamedExpression, to) => UnresolvedAlias(Cast(ne, to))?)
There was a problem hiding this comment.
hmm, how about something like df.select($"a".cast("int").cast("string"))? Do we need to propagate the name to the top level?
There was a problem hiding this comment.
oh, i see. It will be good to propagate the name to the top level. The current change can do the job, right?
There was a problem hiding this comment.
yea, it works, I'll add a test for it.
SparkQA
commented
Jan 18, 2016
Test build #49604 has finished for PR 10781 at commit
|
SparkQA
commented
Jan 18, 2016
Test build #49612 has finished for PR 10781 at commit
|
cloud-fan
commented
Jan 18, 2016
ping @yhuai |
yhuai
commented
Jan 18, 2016
LGTM. Merging to master and branch 1.6. |
yhuai
commented
Jan 18, 2016
there are a few conflicts with branch 1.6. Can you create a pr for the backport? Thanks! |
cloud-fan
commented
Jan 18, 2016
yea, sure |
In SPARK-10743 we wrap cast with `UnresolvedAlias` to give `Cast` a better alias if possible. However, for cases like filter, the `UnresolvedAlias` can't be resolved and actually we don't need a better alias for this case. This PR move the cast wrapping logic to `Column.named` so that we will only do it when we need a alias name. backport #10781 to 1.6 Author: Wenchen Fan <wenchen@databricks.com> Closes#10819 from cloud-fan/bug.
In SPARK-10743 we wrap cast with
UnresolvedAliasto giveCasta better alias if possible. However, for cases likefilter, theUnresolvedAliascan't be resolved and actually we don't need a better alias for this case. This PR move the cast wrapping logic toColumn.namedso that we will only do it when we need a alias name.