Skip to content

[SPARK-21180][SQL] Remove conf from stats functions since now we have conf in LogicalPlan - #18391

Closed
wzhfy wants to merge 3 commits into
apache:masterfrom
wzhfy:removeConf
Closed

[SPARK-21180][SQL] Remove conf from stats functions since now we have conf in LogicalPlan#18391
wzhfy wants to merge 3 commits into
apache:masterfrom
wzhfy:removeConf

Conversation

@wzhfy

@wzhfywzhfy commented Jun 22, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

After wiring SQLConf in logical plan (PR 18299), we can remove the need of passing conf into def stats and def computeStats.

How was this patch tested?

Covered by existing tests, plus some modified existing tests.

@wzhfy

wzhfy commented Jun 22, 2017

Copy link
Copy Markdown
ContributorAuthor

cc @rxin

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala
}

override def computeStats(conf: SQLConf): Statistics =
override def computeStats: Statistics =

@kiszkkiszkJun 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove import org.apache.spark.sql.internal.SQLConf, too?

@wzhfywzhfyJun 23, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked all modified files and removed unused imports. Thanks!

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
))

override def computeStats(conf: SQLConf): Statistics = {
override def computeStats: Statistics = {

@kiszkkiszkJun 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove import org.apache.spark.sql.internal.SQLConf, too?

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
*/
final def stats(conf: SQLConf): Statistics = statsCache.getOrElse {
statsCache = Some(computeStats(conf))
final def stats: Statistics = statsCache.getOrElse {

@kiszkkiszkJun 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

kiszk
kiszk reviewed Jun 22, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala

override def computeStats(conf: SQLConf): Statistics = {
val stats = child.stats(conf)
override def computeStats: Statistics = {

@kiszkkiszkJun 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@SparkQA

SparkQA commented Jun 22, 2017

Copy link
Copy Markdown

Test build #78459 has finished for PR 18391 at commit 5999539.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class FilterEstimation(plan: Filter) extends Logging
  • case class InnerOuterEstimation(join: Join) extends Logging
  • case class LeftSemiAntiEstimation(join: Join)

@kiszk

kiszk commented Jun 23, 2017

Copy link
Copy Markdown
Member

thanks, SGTM

@SparkQA

SparkQA commented Jun 23, 2017

Copy link
Copy Markdown

Test build #78496 has finished for PR 18391 at commit b6025b4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
))

override def computeStats(conf: SQLConf): Statistics = {
override def computeStats: Statistics = {

@gatorsmilegatorsmileJun 23, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this file's import org.apache.spark.sql.internal.SQLConf need to be removed

@kiszkkiszkJun 23, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I overlooked.

@wzhfywzhfyJun 23, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I checked the other interface file, there are several files with this name...

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread...src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/BasicStatsEstimationSuite.scala Outdated
SQLConf.get.setConf(SQLConf.CBO_ENABLED, false)
assert(plan.stats == expectedStatsCboOff)
} finally {
SQLConf.get.unsetConf(SQLConf.CBO_ENABLED)

@gatorsmilegatorsmileJun 23, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first, get the original and then recover the original one in the finally block. You can check how we did it in SQLConfSuite

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread.../src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/AggregateEstimationSuite.scala Outdated
// From UnaryNode.computeStats, childSize * outputRowSize / childRowSize
Statistics(sizeInBytes = 48 * (8 + 4 + 8) / (8 + 4)))
} finally {
SQLConf.get.unsetConf(SQLConf.CBO_ENABLED)

@gatorsmilegatorsmileJun 23, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here.

gatorsmile
gatorsmile reviewed Jun 23, 2017
View reviewed changes
Comment thread...t/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/StatsEstimationTestBase.scala Outdated
}

override def afterAll(): Unit = {
SQLConf.get.unsetConf(SQLConf.CBO_ENABLED)

@gatorsmilegatorsmileJun 23, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here.

@SparkQA

SparkQA commented Jun 23, 2017

Copy link
Copy Markdown

Test build #78521 has finished for PR 18391 at commit cd9f424.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Jun 23, 2017

Copy link
Copy Markdown
Member

LGTM

@gatorsmile

gatorsmile commented Jun 23, 2017

Copy link
Copy Markdown
Member

Thanks! Merging to master.

robert3005 pushed a commit to palantir/spark that referenced this pull request Jun 29, 2017
wangzhenhuaRobert Kruszewski
[SPARK-21180][SQL] Remove conf from stats functions since now we have…
a6296bf
… conf in LogicalPlan
## What changes were proposed in this pull request?
After wiring `SQLConf` in logical plan ([PR 18299](apache#18299)), we can remove the need of passing `conf` into `def stats` and `def computeStats`.
## How was this patch tested?
Covered by existing tests, plus some modified existing tests.
Author: wangzhenhua <wangzhenhua@huawei.com>
Author: Zhenhua Wang <wzh_zju@163.com>
Closesapache#18391 from wzhfy/removeConf.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kiszkkiszkkiszk left review comments
@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wzhfy@SparkQA@kiszk@gatorsmile