Skip to content

SPARK-1338 [scalastyle] Ensure or disallow space after and before token(s). - #284

Closed
ScrapCodes wants to merge 1 commit into
apache:masterfrom
ScrapCodes:SPARK-1338/return-type
Closed

SPARK-1338 [scalastyle] Ensure or disallow space after and before token(s). #284
ScrapCodes wants to merge 1 commit into
apache:masterfrom
ScrapCodes:SPARK-1338/return-type

Conversation

@ScrapCodes

@ScrapCodesScrapCodes commented Apr 1, 2014

Copy link
Copy Markdown
Member

No description provided.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Merged build started.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Merged build finished.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13625/

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Merged build started.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Merged build finished.

@AmplabJenkins

AmplabJenkins commented Apr 1, 2014

Copy link
Copy Markdown

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13628/

ScrapCodes
ScrapCodes reviewed Apr 1, 2014
View reviewed changes
Comment threadproject/spark-style/src/main/scala/org/apache/spark/scalastyle/SpaceAfterColonStyleCheck.scala Outdated

@ScrapCodesScrapCodesApr 1, 2014

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note to self: Rename to SpaceAfterColonChecker.

@rxinrxinApr 1, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this more generic, e.g. SpaceAfterTokenChecker?

@ScrapCodesScrapCodesApr 2, 2014

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can not have any Token checker as colon. Since in case of colon we ensure no space before it and exactly one space after it.
for example

vala:Int=1// validvalb:Int=1//Invalidvalc:Int=1//Invalidvalc:Int=1//Invalid

Also there is a separate check for keywords. ScrapCodes/scalastyle@cccb6b6

@AmplabJenkins

AmplabJenkins commented Apr 9, 2014

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented Apr 9, 2014

Copy link
Copy Markdown

Merged build started.

@AmplabJenkins

AmplabJenkins commented Apr 9, 2014

Copy link
Copy Markdown

Merged build finished. All automated tests passed.

@AmplabJenkins

AmplabJenkins commented Apr 9, 2014

Copy link
Copy Markdown

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13949/

@ScrapCodesScrapCodes changed the title SPARK-1338 scala style for explicitly specified types and short syntax f...SPARK-1338 [scalastyle] Ensure or disallow space after and before token(s). Apr 9, 2014
@ScrapCodes

ScrapCodes commented Jun 5, 2014

Copy link
Copy Markdown
MemberAuthor

This patch is merged in its upstream repo i.e. scalastyle/scalastyle. So if we chose to wait for they making a release, we can actually close this PR. @rxin and @pwendell thoughts ?

@ScrapCodes
ScrapCodes deleted the SPARK-1338/return-type branch June 3, 2015 06:01
rahij pushed a commit to rahij/spark that referenced this pull request Dec 5, 2017
@ash211
Merge pull requestapache#284from palantir/aash/resync-apache
1c1a608
[NOSQUASH] Resync from Apache
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
@kiwik
Merge pull requestapache#284from liu-sheng/fix-packer-devstack-mitaka
97b0fe4
Add OS_DOMAIN_ID env for Packer job with Mitaka devstack
wangyum added a commit that referenced this pull request Aug 19, 2021
@wangyum
[SPARK-36444][SQL] Remove OptimizeSubqueries from batch of PartitionP…
2310b99
…runing
### What changes were proposed in this pull request?
Remove `OptimizeSubqueries` from batch of `PartitionPruning` to make DPP support more cases. For example:
```sql
SELECT date_id, product_id FROM fact_sk f
JOIN (select store_id + 3 as new_store_id from dim_store where country = 'US') s
ON f.store_id = s.new_store_id
```
Before this PR:
```
== Physical Plan ==
*(2) Project [date_id#3998, product_id#3999]
+- *(2) BroadcastHashJoin [store_id#4001], [new_store_id#3997], Inner, BuildRight, false
:- *(2) ColumnarToRow
: +- FileScan parquet default.fact_sk[date_id#3998,product_id#3999,store_id#4001] Batched: true, DataFilters: [], Format: Parquet, PartitionFilters: [isnotnull(store_id#4001), dynamicpruningexpression(true)], PushedFilters: [], ReadSchema: struct<date_id:int,product_id:int>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#274]
+- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
+- *(1) Filter ((isnotnull(country#4004) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
+- *(1) ColumnarToRow
+- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002 + 3))], Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
```
After this PR:
```
== Physical Plan ==
*(2) Project [date_id#3998, product_id#3999]
+- *(2) BroadcastHashJoin [store_id#4001], [new_store_id#3997], Inner, BuildRight, false
:- *(2) ColumnarToRow
: +- FileScan parquet default.fact_sk[date_id#3998,product_id#3999,store_id#4001] Batched: true, DataFilters: [], Format: Parquet, PartitionFilters: [isnotnull(store_id#4001), dynamicpruningexpression(store_id#4001 IN dynamicpruning#4007)], PushedFilters: [], ReadSchema: struct<date_id:int,product_id:int>
: +- SubqueryBroadcast dynamicpruning#4007, 0, [new_store_id#3997], [id=#263]
: +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#262]
: +- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
: +- *(1) Filter ((isnotnull(country#4004) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
: +- *(1) ColumnarToRow
: +- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002 + 3))], Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
+- ReusedExchange [new_store_id#3997], BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#262]
```
This is because `OptimizeSubqueries` will infer more filters, so we cannot reuse broadcasts. The following is the plan if disable `spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly`:
```
== Physical Plan ==
*(2) Project [date_id#3998, product_id#3999]
+- *(2) BroadcastHashJoin [store_id#4001], [new_store_id#3997], Inner, BuildRight, false
:- *(2) ColumnarToRow
: +- FileScan parquet default.fact_sk[date_id#3998,product_id#3999,store_id#4001] Batched: true, DataFilters: [], Format: Parquet, PartitionFilters: [isnotnull(store_id#4001), dynamicpruningexpression(store_id#4001 IN subquery#4009)], PushedFilters: [], ReadSchema: struct<date_id:int,product_id:int>
: +- Subquery subquery#4009, [id=#284]
: +- *(2) HashAggregate(keys=[new_store_id#3997#4008], functions=[])
: +- Exchange hashpartitioning(new_store_id#3997#4008, 5), ENSURE_REQUIREMENTS, [id=#280]
: +- *(1) HashAggregate(keys=[new_store_id#3997 AS new_store_id#3997#4008], functions=[])
: +- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
: +- *(1) Filter (((isnotnull(store_id#4002) AND isnotnull(country#4004)) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
: +- *(1) ColumnarToRow
: +- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(store_id#4002), isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002..., Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(store_id), IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#305]
+- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
+- *(1) Filter ((isnotnull(country#4004) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
+- *(1) ColumnarToRow
+- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002 + 3))], Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
```
### Why are the changes needed?
Improve DPP to support more cases.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and benchmark test:
SQL | Before this PR(Seconds) | After this PR(Seconds)
-- | -- | --
TPC-DS q58 | 40 | 20
TPC-DS q83 | 18 | 14
Closes#33664 from wangyum/SPARK-36444.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
wangyum added a commit that referenced this pull request Aug 19, 2021
@wangyum
[SPARK-36444][SQL] Remove OptimizeSubqueries from batch of PartitionP…
5b97165
…runing
### What changes were proposed in this pull request?
Remove `OptimizeSubqueries` from batch of `PartitionPruning` to make DPP support more cases. For example:
```sql
SELECT date_id, product_id FROM fact_sk f
JOIN (select store_id + 3 as new_store_id from dim_store where country = 'US') s
ON f.store_id = s.new_store_id
```
Before this PR:
```
== Physical Plan ==
*(2) Project [date_id#3998, product_id#3999]
+- *(2) BroadcastHashJoin [store_id#4001], [new_store_id#3997], Inner, BuildRight, false
:- *(2) ColumnarToRow
: +- FileScan parquet default.fact_sk[date_id#3998,product_id#3999,store_id#4001] Batched: true, DataFilters: [], Format: Parquet, PartitionFilters: [isnotnull(store_id#4001), dynamicpruningexpression(true)], PushedFilters: [], ReadSchema: struct<date_id:int,product_id:int>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#274]
+- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
+- *(1) Filter ((isnotnull(country#4004) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
+- *(1) ColumnarToRow
+- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002 + 3))], Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
```
After this PR:
```
== Physical Plan ==
*(2) Project [date_id#3998, product_id#3999]
+- *(2) BroadcastHashJoin [store_id#4001], [new_store_id#3997], Inner, BuildRight, false
:- *(2) ColumnarToRow
: +- FileScan parquet default.fact_sk[date_id#3998,product_id#3999,store_id#4001] Batched: true, DataFilters: [], Format: Parquet, PartitionFilters: [isnotnull(store_id#4001), dynamicpruningexpression(store_id#4001 IN dynamicpruning#4007)], PushedFilters: [], ReadSchema: struct<date_id:int,product_id:int>
: +- SubqueryBroadcast dynamicpruning#4007, 0, [new_store_id#3997], [id=#263]
: +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#262]
: +- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
: +- *(1) Filter ((isnotnull(country#4004) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
: +- *(1) ColumnarToRow
: +- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002 + 3))], Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
+- ReusedExchange [new_store_id#3997], BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#262]
```
This is because `OptimizeSubqueries` will infer more filters, so we cannot reuse broadcasts. The following is the plan if disable `spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly`:
```
== Physical Plan ==
*(2) Project [date_id#3998, product_id#3999]
+- *(2) BroadcastHashJoin [store_id#4001], [new_store_id#3997], Inner, BuildRight, false
:- *(2) ColumnarToRow
: +- FileScan parquet default.fact_sk[date_id#3998,product_id#3999,store_id#4001] Batched: true, DataFilters: [], Format: Parquet, PartitionFilters: [isnotnull(store_id#4001), dynamicpruningexpression(store_id#4001 IN subquery#4009)], PushedFilters: [], ReadSchema: struct<date_id:int,product_id:int>
: +- Subquery subquery#4009, [id=#284]
: +- *(2) HashAggregate(keys=[new_store_id#3997#4008], functions=[])
: +- Exchange hashpartitioning(new_store_id#3997#4008, 5), ENSURE_REQUIREMENTS, [id=#280]
: +- *(1) HashAggregate(keys=[new_store_id#3997 AS new_store_id#3997#4008], functions=[])
: +- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
: +- *(1) Filter (((isnotnull(store_id#4002) AND isnotnull(country#4004)) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
: +- *(1) ColumnarToRow
: +- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(store_id#4002), isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002..., Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(store_id), IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#305]
+- *(1) Project [(store_id#4002 + 3) AS new_store_id#3997]
+- *(1) Filter ((isnotnull(country#4004) AND (country#4004 = US)) AND isnotnull((store_id#4002 + 3)))
+- *(1) ColumnarToRow
+- FileScan parquet default.dim_store[store_id#4002,country#4004] Batched: true, DataFilters: [isnotnull(country#4004), (country#4004 = US), isnotnull((store_id#4002 + 3))], Format: Parquet, PartitionFilters: [], PushedFilters: [IsNotNull(country), EqualTo(country,US)], ReadSchema: struct<store_id:int,country:string>
```
### Why are the changes needed?
Improve DPP to support more cases.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and benchmark test:
SQL | Before this PR(Seconds) | After this PR(Seconds)
-- | -- | --
TPC-DS q58 | 40 | 20
TPC-DS q83 | 18 | 14
Closes#33664 from wangyum/SPARK-36444.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
(cherry picked from commit 2310b99)
Signed-off-by: Yuming Wang <yumwang@ebay.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ScrapCodes@AmplabJenkins@rxin