[SPARK-12275][SQL] No plan for BroadcastHint in some condition by yucai · Pull Request #10265 · apache/spark

yucai · 2015-12-11T05:45:24Z

When SparkStrategies.BasicOperators's "case BroadcastHint(child) => apply(child)" is hit, it only recursively invokes BasicOperators.apply with this "child". It makes many strategies have no change to process this plan, which probably leads to "No plan" issue, so we use planLater to go through all strategies.

https://issues.apache.org/jira/browse/SPARK-12275

zzcclp · 2015-12-11T06:19:42Z

Hi, @yucai , my code is executed successfully with this pr. It's great.

yucai · 2015-12-11T06:40:32Z

@zzcclp Good to know :).

chenghao-intel · 2015-12-11T09:32:08Z

LGTM
cc/ @yhuai

andrewor14 · 2015-12-11T21:29:18Z

ok to test. @yhuai can you verify?

SparkQA · 2015-12-11T23:13:27Z

Test build #47592 has finished for PR 10265 at commit 1b8d570.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2015-12-12T00:14:16Z

The failure seems not related to this change.

chenghao-intel · 2015-12-12T00:23:11Z

This is quite obvious, apply method will only make the rules within the object BasicOperators applied, but the data source(Parquet) is an extended strategy defined in another object, planLater means to search the all of the strategies(including the extensions), which is make more sense. That's why it caused exception says No plan for BroadcastHint as apply(child) returns Nil.

yucai · 2015-12-13T02:47:40Z

Weird, I find no difference for org.apache.spark.sql.sources.JsonHadoopFsRelationSuite#"test all data types - TimestampType" in Jekins's test report.

And also, I tried both org.apache.spark.sql.sources.JsonHadoopFsRelationSuite and org.apache.spark.sql.DataFrameJoinSuite with the latest Spark upstream + my PR locally, no failure also.
The executed command is:

build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Phive-thriftserver -Phive -DskipTests assembly
build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Phive-thriftserver -Phive 'test-only org.apache.spark.sql.sources.JsonHadoopFsRelationSuite'
build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Phive-thriftserver -Phive 'test-only org.apache.spark.sql.DataFrameJoinSuite'

What's wrong :( ?

Jekins, can you retest this please?

yucai · 2015-12-14T02:34:08Z

retest this please

yucai · 2015-12-14T04:22:31Z

@andrewor14 , @yhuai could you kindly help re-trigger testing? Much thanks!

yhuai · 2015-12-14T05:15:23Z

ok to test

yhuai · 2015-12-14T05:15:28Z

test this please

yhuai · 2015-12-14T05:20:11Z

LGTM pending jenkins

yhuai · 2015-12-14T05:21:00Z

When merge this, let's also merge it to branch 1.5, 1.6, and master.

SparkQA · 2015-12-14T06:57:25Z

Test build #47642 has finished for PR 10265 at commit 1b8d570.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2015-12-14T07:06:11Z

Thanks! Merging to 1.5, 1.6, and master.

When SparkStrategies.BasicOperators's "case BroadcastHint(child) => apply(child)" is hit, it only recursively invokes BasicOperators.apply with this "child". It makes many strategies have no change to process this plan, which probably leads to "No plan" issue, so we use planLater to go through all strategies. https://issues.apache.org/jira/browse/SPARK-12275 Author: yucai <yucai.yu@intel.com> Closes#10265 from yucai/broadcast_hint. (cherry picked from commit ed87f6d) Signed-off-by: Yin Huai <yhuai@databricks.com>

yhuai · 2015-12-14T07:13:26Z

@yucai can you create a 1.5 backport for this one? There was a conflict when I tried to merge it to branch 1.5. It will be good to have a pr for the backport and make sure all tests are good. Thanks!

yucai · 2015-12-14T07:15:11Z

@yhuai sure, I will do that ASAP.

…backport backport apache#10265 to branch 1.5 When SparkStrategies.BasicOperators's "case BroadcastHint(child) => apply(child)" is hit, it only recursively invokes BasicOperators.apply with this "child". It makes many strategies have no change to process this plan, which probably leads to "No plan" issue, so we use planLater to go through all strategies. https://issues.apache.org/jira/browse/SPARK-12275

… backport backport #10265 to branch 1.5. When SparkStrategies.BasicOperators's "case BroadcastHint(child) => apply(child)" is hit, it only recursively invokes BasicOperators.apply with this "child". It makes many strategies have no change to process this plan, which probably leads to "No plan" issue, so we use planLater to go through all strategies. https://issues.apache.org/jira/browse/SPARK-12275 Author: yucai <yucai.yu@intel.com> Closes#10291 from yucai/backport_1.5_no_plan_for_broadcasthint and squashes the following commits: b09715c [yucai] [SPARK-12275][SQL] No plan for BroadcastHint in some condition - 1.5 backport

yucai added 2 commits December 11, 2015 06:41

[SPARK-12275][SQL] No plan for BroadcastHint in some condition
8ced744

Add test cases
1b8d570

chenghao-intel reviewed Dec 12, 2015
View reviewed changes

asfgit closed this in ed87f6dDec 14, 2015

yucai mentioned this pull request Dec 14, 2015
[SPARK-12275][SQL] No plan for BroadcastHint in some condition - 1.5 backport #10291
Closed

Uh oh!

Conversation

yucai commented Dec 11, 2015

Uh oh!

zzcclp commented Dec 11, 2015

Uh oh!

yucai commented Dec 11, 2015

Uh oh!

chenghao-intel commented Dec 11, 2015

Uh oh!

andrewor14 commented Dec 11, 2015

Uh oh!

SparkQA commented Dec 11, 2015

Uh oh!

chenghao-intel commented Dec 12, 2015

Uh oh!

chenghao-intelDec 12, 2015

Choose a reason for hiding this comment

Uh oh!

yucai commented Dec 13, 2015

Uh oh!

yucai commented Dec 14, 2015

Uh oh!

yucai commented Dec 14, 2015

Uh oh!

yhuai commented Dec 14, 2015

Uh oh!

yhuai commented Dec 14, 2015

Uh oh!

yhuai commented Dec 14, 2015

Uh oh!

yhuai commented Dec 14, 2015

Uh oh!

SparkQA commented Dec 14, 2015

Uh oh!

yhuai commented Dec 14, 2015

Uh oh!

yhuai commented Dec 14, 2015

Uh oh!

yucai commented Dec 14, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants