[SPARK-2393][SQL] Cost estimation optimization framework for Catalyst logical plans & sample usage. by concretevitamin · Pull Request #1238 · apache/spark

concretevitamin · 2014-06-26T23:14:33Z

The idea is that every Catalyst logical plan gets hold of a Statistics class, the usage of which provides useful estimations on various statistics. See the implementations of MetastoreRelation.

This patch also includes several usages of the estimation interface in the planner. For instance, we now use physical table sizes from the estimate interface to convert an equi-join to a broadcast join (when doing so is beneficial, as determined by a size threshold).

Finally, there are a couple minor accompanying changes including:

Remove the not-in-use BaseRelation.
Make SparkLogicalPlan take a SQLContext in the second param list.

AmplabJenkins · 2014-06-26T23:15:24Z

Merged build triggered.

AmplabJenkins · 2014-06-26T23:15:34Z

Merged build started.

AmplabJenkins · 2014-06-26T23:20:24Z

Merged build triggered.

AmplabJenkins · 2014-06-26T23:20:34Z

Merged build started.

AmplabJenkins · 2014-06-26T23:22:10Z

Merged build finished.

AmplabJenkins · 2014-06-26T23:22:10Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16174/

AmplabJenkins · 2014-06-26T23:25:38Z

Merged build finished.

AmplabJenkins · 2014-06-26T23:25:39Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16173/

rxin · 2014-06-26T23:45:26Z

remove broadcastTables.contains(b.tableName) and then your if can fit in a single line, and you don't need to indent the next line ...

AmplabJenkins · 2014-07-03T21:25:49Z

Merged build triggered.

AmplabJenkins · 2014-07-03T21:25:57Z

Merged build started.

AmplabJenkins · 2014-07-03T21:30:49Z

Merged build triggered.

AmplabJenkins · 2014-07-03T21:30:57Z

Merged build started.

AmplabJenkins · 2014-07-03T21:36:01Z

Merged build finished.

AmplabJenkins · 2014-07-03T21:36:02Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16316/

concretevitamin · 2014-07-03T22:03:32Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-03T22:05:49Z

Merged build triggered.

AmplabJenkins · 2014-07-03T22:05:57Z

Merged build started.

AmplabJenkins · 2014-07-03T22:16:02Z

Merged build finished.

AmplabJenkins · 2014-07-03T22:16:02Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16319/

AmplabJenkins · 2014-07-03T22:51:23Z

Merged build finished.

AmplabJenkins · 2014-07-03T22:51:23Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16317/

AmplabJenkins · 2014-07-03T22:55:50Z

Merged build triggered.

AmplabJenkins · 2014-07-03T22:55:58Z

Merged build started.

Additionally, remove size estimate from ParquetRelation since the Hadoop FileSystem API calls can be expensive (e.g. S3FileSystem has a lot of RPCs).

Also cleanups & scaladoc fixes per review comments.

…Conf.

concretevitamin · 2014-07-29T19:00:48Z

Rebased & addressed review comments.

concretevitamin · 2014-07-29T19:00:55Z

Jenkins test this please

SparkQA · 2014-07-29T19:04:08Z

QA tests have started for PR 1238. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17369/consoleFull

SparkQA · 2014-07-29T20:16:12Z

QA results for PR 1238:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17369/consoleFull

marmbrus · 2014-07-29T22:33:11Z

Thanks! I've merged this into master.

@PARAM

… logical plans & sample usage. The idea is that every Catalyst logical plan gets hold of a Statistics class, the usage of which provides useful estimations on various statistics. See the implementations of `MetastoreRelation`. This patch also includes several usages of the estimation interface in the planner. For instance, we now use physical table sizes from the estimate interface to convert an equi-join to a broadcast join (when doing so is beneficial, as determined by a size threshold). Finally, there are a couple minor accompanying changes including: - Remove the not-in-use `BaseRelation`. - Make SparkLogicalPlan take a `SQLContext` in the second param list. Author: Zongheng Yang <zongheng.y@gmail.com> Closesapache#1238 from concretevitamin/estimates and squashes the following commits: 329071d [Zongheng Yang] Address review comments; turn config name from string to field in SQLConf. 8663e84 [Zongheng Yang] Use BigInt for stat; for logical leaves, by default throw an exception. 2f2fb89 [Zongheng Yang] Fix statistics for SparkLogicalPlan. 9951305 [Zongheng Yang] Remove childrenStats. 16fc60a [Zongheng Yang] Avoid calling statistics on plans if auto join conversion is disabled. 8bd2816 [Zongheng Yang] Add a note on performance of statistics. 6e594b8 [Zongheng Yang] Get size info from metastore for MetastoreRelation. 01b7a3e [Zongheng Yang] Update scaladoc for a field and move it to @PARAM section. 549061c [Zongheng Yang] Remove numTuples in Statistics for now. 729a8e2 [Zongheng Yang] Update docs to be more explicit. 573e644 [Zongheng Yang] Remove singleton SQLConf and move back `settings` to the trait. 2d99eb5 [Zongheng Yang] {Cleanup, use synchronized in, enrich} StatisticsSuite. ca5b825 [Zongheng Yang] Inject SQLContext into SparkLogicalPlan, removing SQLConf mixin from it. 43d38a6 [Zongheng Yang] Revert optimization for BroadcastNestedLoopJoin (this fixes tests). 0ef9e5b [Zongheng Yang] Use multiplication instead of sum for default estimates. 4ef0d26 [Zongheng Yang] Make Statistics a case class. 3ba8f3e [Zongheng Yang] Add comment. e5bcf5b [Zongheng Yang] Fix optimization conditions & update scala docs to explain. 7d9216a [Zongheng Yang] Apply estimation to planning ShuffleHashJoin & BroadcastNestedLoopJoin. 73cde01 [Zongheng Yang] Move SQLConf back. Assign default sizeInBytes to SparkLogicalPlan. 73412be [Zongheng Yang] Move SQLConf to Catalyst & add default val for sizeInBytes. 7a60ab7 [Zongheng Yang] s/Estimates/Statistics, s/cardinality/numTuples. de3ae13 [Zongheng Yang] Add parquetAfter() properly in test. dcff9bd [Zongheng Yang] Cleanups. 84301a4 [Zongheng Yang] Refactors. 5bf5586 [Zongheng Yang] Typo. 56a8e6e [Zongheng Yang] Prototype impl of estimations for Catalyst logical plans.

* [CARMEL-6439] Add configuration to enable log column lineage * remove log * useless condition * skip empty lineage

rxin reviewed Jun 26, 2014
View reviewed changes

concretevitamin added 18 commits July 29, 2014 11:48

Fix optimization conditions & update scala docs to explain.
e5bcf5b

Add comment.
3ba8f3e

Make Statistics a case class.
4ef0d26

Use multiplication instead of sum for default estimates.
0ef9e5b

Revert optimization for BroadcastNestedLoopJoin (this fixes tests).
43d38a6

Inject SQLContext into SparkLogicalPlan, removing SQLConf mixin from it.
ca5b825

{Cleanup, use synchronized in, enrich} StatisticsSuite.
2d99eb5

Remove singleton SQLConf and move back settings to the trait.
573e644

Update docs to be more explicit.
729a8e2

Remove numTuples in Statistics for now.
549061c

Update scaladoc for a field and move it to@PARAMsection.
01b7a3e

Get size info from metastore for MetastoreRelation.
6e594b8
Additionally, remove size estimate from ParquetRelation since the Hadoop FileSystem API calls can be expensive (e.g. S3FileSystem has a lot of RPCs).

Add a note on performance of statistics.
8bd2816

Avoid calling statistics on plans if auto join conversion is disabled.
16fc60a

Remove childrenStats.
9951305

Fix statistics for SparkLogicalPlan.
2f2fb89

Use BigInt for stat; for logical leaves, by default throw an exception.
8663e84
Also cleanups & scaladoc fixes per review comments.

Address review comments; turn config name from string to field in SQL…
329071d
…Conf.

asfgit closed this in c7db274Jul 29, 2014

wangyum pushed a commit that referenced this pull request May 26, 2023
[CARMEL-6439] Add configuration to enable log column lineage (#1238)
9f306b4
* [CARMEL-6439] Add configuration to enable log column lineage * remove log * useless condition * skip empty lineage

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025
MapR [SPARK-1340] JWT support at Spark (apache#1238)
18c2259

Uh oh!

Conversation

concretevitamin commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

rxinJun 26, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

concretevitamin commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 3, 2014

Uh oh!

concretevitamin commented Jul 29, 2014

Uh oh!

concretevitamin commented Jul 29, 2014

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

marmbrus commented Jul 29, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants