Skip to content

[SPARK-20413] Add new query hint NO_COLLAPSE. - #17708

Closed
ptkool wants to merge 2 commits into
apache:masterfrom
ptkool:no_collapse_query_hint
Closed

[SPARK-20413] Add new query hint NO_COLLAPSE.#17708
ptkool wants to merge 2 commits into
apache:masterfrom
ptkool:no_collapse_query_hint

Conversation

@ptkool

@ptkoolptkool commented Apr 20, 2017

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR proposes adding a new query hint called NO_COLLAPSE that can be used to prevent adjacent projections from being collapsed.

How was this patch tested?

Test using unit tests, integration tests and manual tests.

@ptkoolptkool changed the title Add new query hint NO_COLLAPSE.[SPARK-20413] Add new query hint NO_COLLAPSE.Apr 20, 2017
@hvanhovell

hvanhovell commented Apr 20, 2017

Copy link
Copy Markdown
Contributor

ok to test

@SparkQA

SparkQA commented Apr 20, 2017

Copy link
Copy Markdown

Test build #75995 has finished for PR 17708 at commit 3f1e6a1.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class NoCollapseHint(child: LogicalPlan) extends UnaryNode

hvanhovell
hvanhovell requested changes Apr 20, 2017
View reviewed changes

@hvanhovellhvanhovell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ptkool thinks for submitting the PR. I am not sure this is the best way to avoid projection collapse. The problem is that this approach will also inhibit other optimization from taking place.

Comment threadpython/pyspark/sql/functions.py Outdated

@since(2.2)
def no_collapse(df):
"""Marks a DataFrame as small enough for use in broadcast joins."""

@hvanhovellhvanhovellApr 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc is incorrect.

@ptkoolptkoolApr 20, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fix.

Comment threadsql/core/src/main/scala/org/apache/spark/sql/functions.scala Outdated
* @group normal_funcs
* @since 1.5.0
*/
* Marks a DataFrame as small enough for use in broadcast joins.

@hvanhovellhvanhovellApr 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please undo this change.

@ptkoolptkoolApr 20, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

Comment threadsql/core/src/main/scala/org/apache/spark/sql/functions.scala Outdated
}

/**
* Marks a DataFrame as small enough for use in broadcast joins.

@hvanhovellhvanhovellApr 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the alignment is of by a space, it should be:

/**
* Text...
*/

@ptkoolptkoolApr 20, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fix.

Comment thread...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

/**
* A hint for the optimizer that we should not merge two projections.
*/

@hvanhovellhvanhovellApr 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why we want this in the LogicalPlan level and not on the expression level?

@hvanhovellhvanhovellApr 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with this approach is that most other optimizations won't work with this, for example predicate push down.

@ptkoolptkoolApr 20, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally thought about putting it at the expression level, but ultimately decided it made more sense at the LogicalPlan node level, since the purpose was in fact to disrupt the optimizer. In some respects, it's meant to have the same effect as df.cache(), but without the caching. There may, in fact, be situations where predicate pushdown is not desired because the resulting condition would become complex and expensive to evaluate.

In Spark SQL, I think it also makes more sense to specify the hint at the derived table level, as opposed to a single expression. For instance,

SELECT SNO, PNO, C1 +1, C1 + 2
FROM ( SELECT /*+ NO_COLLAPSE */ SNO, PNO, QTY * 10 AS C1 FROM T ) T

This is similar to the NO_MERGE query hint in Oracle, which prevents the query from being flattened.

Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala Outdated

comparePlans(
parsePlan("SELECT a FROM (SELECT /*+ NO_COLLAPSE */ * FROM t) t1"),
SubqueryAlias("t1", Hint("NO_COLLAPSE", Seq.empty, table("t").select(star())))

@hvanhovellhvanhovellApr 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are you testing here that is not covered by the other cases?

@ptkoolptkoolApr 20, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, nothing. I will remove it.

@SparkQA

SparkQA commented Apr 20, 2017

Copy link
Copy Markdown

Test build #76001 has finished for PR 17708 at commit 975cca5.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ptkool
ptkoolforce-pushed the no_collapse_query_hint branch from 1231585 to 3986247CompareApril 20, 2017 20:19
@SparkQA

SparkQA commented Apr 20, 2017

Copy link
Copy Markdown

Test build #76005 has finished for PR 17708 at commit 3986247.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Apr 23, 2017•
edited
Loading

Copy link
Copy Markdown
Member

Based on the JIRA description, it sounds like we should not simply merge two Projects to avoid calling the same UDF multiple times, instead of adding a new logical plan node.

@viirya

viirya commented Apr 24, 2017

Copy link
Copy Markdown
Member

I have the same question as Reynold asked in the mailing list. Doesn't common sub expression elimination already address this issue?

@gatorsmile

gatorsmile commented Jun 14, 2017

Copy link
Copy Markdown
Member

Any update? Maybe we can close this PR at first?

@HyukjinKwonHyukjinKwon mentioned this pull request Jun 25, 2017
Closed
@ptkool

ptkool commented Jun 26, 2017

Copy link
Copy Markdown
ContributorAuthor

@gatorsmile I will run a few more tests to determine if subexpression elimination solves this issue.

@gatorsmile

gatorsmile commented Jun 27, 2017

Copy link
Copy Markdown
Member

We are closing the inactive PRs. After you run more test, please do reopen if you still hit this issue. Thanks!

zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PRs
ddbe288
## What changes were proposed in this pull request?
This PR proposes to close stale PRs, mostly the same instances with apache#18017
I believe the author in apache#14807 removed his account.
Closesapache#7075Closesapache#8927Closesapache#9202Closesapache#9366Closesapache#10861Closesapache#11420Closesapache#12356Closesapache#13028Closesapache#13506Closesapache#14191Closesapache#14198Closesapache#14330Closesapache#14807Closesapache#15839Closesapache#16225Closesapache#16685Closesapache#16692Closesapache#16995Closesapache#17181Closesapache#17211Closesapache#17235Closesapache#17237Closesapache#17248Closesapache#17341Closesapache#17708Closesapache#17716Closesapache#17721Closesapache#17937
Added:
Closesapache#14739Closesapache#17139Closesapache#17445Closesapache#18042Closesapache#18359
Added:
Closesapache#16450Closesapache#16525Closesapache#17738
Added:
Closesapache#16458Closesapache#16508Closesapache#17714
Added:
Closesapache#17830Closesapache#14742
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18417 from HyukjinKwon/close-stale-pr.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hvanhovellhvanhovellhvanhovell requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ptkool@hvanhovell@SparkQA@gatorsmile@viirya