Skip to content

[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unresolved state - #20914

Closed
robert3005 wants to merge 6 commits into
apache:masterfrom
robert3005:rk/propagate-empty-fix
Closed

[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unresolved state#20914
robert3005 wants to merge 6 commits into
apache:masterfrom
robert3005:rk/propagate-empty-fix

Conversation

@robert3005

@robert3005robert3005 commented Mar 27, 2018

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules

How was this patch tested?

Added unit test

@hvanhovell

hvanhovell commented Mar 27, 2018

Copy link
Copy Markdown
Contributor

ok to test

@SparkQA

SparkQA commented Mar 27, 2018

Copy link
Copy Markdown

Test build #88630 has finished for PR 20914 at commit be950e4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@robert3005

robert3005 commented Mar 27, 2018

Copy link
Copy Markdown
ContributorAuthor

org.apache.spark.sql.execution.streaming.RateSourceV2Suite.basic microbatch execution failed which looks like a flake to me

@hvanhovell

hvanhovell commented Mar 27, 2018

Copy link
Copy Markdown
Contributor

retest this please

@SparkQA

SparkQA commented Mar 28, 2018

Copy link
Copy Markdown

Test build #88642 has finished for PR 20914 at commit be950e4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Mar 28, 2018

Copy link
Copy Markdown
Member

retest this please

@SparkQA

SparkQA commented Mar 28, 2018

Copy link
Copy Markdown

Test build #88646 has finished for PR 20914 at commit be950e4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@robert3005
robert3005force-pushed the rk/propagate-empty-fix branch from be950e4 to d9a2ee8CompareMarch 29, 2018 11:37
@SparkQA

SparkQA commented Mar 29, 2018

Copy link
Copy Markdown

Test build #88705 has finished for PR 20914 at commit d9a2ee8.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

robert3005
robert3005 commented Mar 29, 2018
View reviewed changes
Comment thread...yst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala Outdated
.where(left)
.join(testRelation2.where(right), joinType = jt, condition = Some('a.attr == 'b.attr))
val optimized = Optimize.execute(query.analyze)
val optimized = Optimize.execute(query.analyze).analyze

@robert3005robert3005Mar 29, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the subsequent analyze here is to fill in timezones on casts inserted by Optimize

@gatorsmilegatorsmileMar 30, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should fill the time zone in the optimizer rule

@SparkQA

SparkQA commented Mar 29, 2018

Copy link
Copy Markdown

Test build #88717 has finished for PR 20914 at commit cd96401.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed Mar 30, 2018
View reviewed changes
Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala Outdated
// Construct a project list from plan's output, while the value is always NULL.
private def nullValueProjectList(plan: LogicalPlan): Seq[NamedExpression] =
plan.output.map{ a => Alias(Literal(null), a.name)(a.exprId) }
plan.output.map{ a => Alias(Cast(Literal(null), a.dataType), a.name)(a.exprId) }

@gatorsmilegatorsmileMar 30, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use CastSupport.cast

@SparkQA

SparkQA commented Mar 30, 2018

Copy link
Copy Markdown

Test build #88759 has finished for PR 20914 at commit 1c123bd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@robert3005

robert3005 commented Apr 3, 2018

Copy link
Copy Markdown
ContributorAuthor

@gatorsmile how does it look now?

@gatorsmile

gatorsmile commented Apr 4, 2018

Copy link
Copy Markdown
Member

LGTM

asfgit pushed a commit that referenced this pull request Apr 4, 2018
Robert Kruszewski@gatorsmile
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unr…
28c9adb
…esolved state
## What changes were proposed in this pull request?
Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules
## How was this patch tested?
Added unit test
Author: Robert Kruszewski <robertk@palantir.com>
Closes#20914 from robert3005/rk/propagate-empty-fix.
(cherry picked from commit 5cfd5fa)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
@asfgitasfgit closed this in 5cfd5faApr 4, 2018
robert3005 pushed a commit to palantir/spark that referenced this pull request Apr 4, 2018
Robert Kruszewski
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unr…
150b653
…esolved state
## What changes were proposed in this pull request?
Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules
## How was this patch tested?
Added unit test
Author: Robert Kruszewski <robertk@palantir.com>
Closesapache#20914 from robert3005/rk/propagate-empty-fix.
mshtelma pushed a commit to mshtelma/spark that referenced this pull request Apr 5, 2018
Robert KruszewskiMykhailo Shtelma
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unr…
b55e39f
…esolved state
## What changes were proposed in this pull request?
Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules
## How was this patch tested?
Added unit test
Author: Robert Kruszewski <robertk@palantir.com>
Closesapache#20914 from robert3005/rk/propagate-empty-fix.
peter-toth pushed a commit to peter-toth/spark that referenced this pull request Oct 6, 2018
Robert KruszewskiHyukjin Kwon
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unr…
7e33486
…esolved state
Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules
Added unit test
Author: Robert Kruszewski <robertk@palantir.com>
Closesapache#20914 from robert3005/rk/propagate-empty-fix.
(cherry picked from commit 5cfd5fa)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
Change-Id: I9c673e572454a9fa43e296b7fe66e7f2fc569854
@robert3005
robert3005 deleted the rk/propagate-empty-fix branch December 6, 2019 12:49
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@robert3005@hvanhovell@SparkQA@gatorsmile