Skip to content

[SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite - #33510

Closed
roryqi wants to merge 12 commits into
apache:masterfrom
roryqi:SPARK-36223
Closed

[SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite#33510
roryqi wants to merge 12 commits into
apache:masterfrom
roryqi:SPARK-36223

Conversation

@roryqi

@roryqiroryqi commented Jul 24, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

In current github actions we run TPCDSQueryTestSuite for tpcds benchmark. But it's only tested under default configurations. Since we have added the spark.sql.join.forceApplyShuffledHashJoin config. Now we can test all 3 join strategies in TPCDS to improve the coverage.

Why are the changes needed?

Improve the coverage of join strategies in the TPCDS.

Does this PR introduce any user-facing change?

No, only for testing.

How was this patch tested?

No need.

@roryqiroryqi changed the title [SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite[WIP][SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuiteJul 25, 2021
@roryqi
roryqiforce-pushed the SPARK-36223 branch 7 times, most recently from a603ffc to 46bbf1fCompareJuly 25, 2021 16:11
@roryqiroryqi changed the title [WIP][SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuite[SPARK-36223][SQL][TEST] Cover 3 kinds of join in the TPCDSQueryTestSuiteJul 25, 2021
linhongliu-db
linhongliu-db reviewed Jul 29, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
linhongliu-db
linhongliu-db reviewed Jul 29, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
@roryqi
roryqiforce-pushed the SPARK-36223 branch 4 times, most recently from 42bb832 to 763c492CompareAugust 6, 2021 13:58
cloud-fan
cloud-fan reviewed Nov 10, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
cloud-fan
cloud-fan reviewed Nov 10, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
@cloud-fan

cloud-fan commented Nov 10, 2021

Copy link
Copy Markdown
Contributor

ok to test

@SparkQA

SparkQA commented Nov 10, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49535/

@SparkQA

SparkQA commented Nov 10, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49535/

@SparkQA

SparkQA commented Nov 10, 2021

Copy link
Copy Markdown

Test build #145066 has finished for PR 33510 at commit 8e2f4fb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

HyukjinKwon
HyukjinKwon reviewed Nov 11, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
HyukjinKwon
HyukjinKwon reviewed Nov 11, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala Outdated
@SparkQA

SparkQA commented Nov 11, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49589/

@SparkQA

SparkQA commented Nov 11, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49590/

@SparkQA

SparkQA commented Nov 11, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49589/

@SparkQA

SparkQA commented Nov 11, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49590/

@SparkQA

SparkQA commented Nov 12, 2021

Copy link
Copy Markdown

Test build #145167 has finished for PR 33510 at commit cb0d3bc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Nov 12, 2021

Copy link
Copy Markdown

Test build #145170 has finished for PR 33510 at commit cf38cab.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Nov 15, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49692/

@SparkQA

SparkQA commented Nov 15, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49692/

@SparkQA

SparkQA commented Nov 15, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49693/

@cloud-fan

cloud-fan commented Nov 15, 2021

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@SparkQA

SparkQA commented Nov 15, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49693/

@roryqi

roryqi commented Nov 15, 2021

Copy link
Copy Markdown
ContributorAuthor

Thank you @cloud-fan@linhongliu-db and @HyukjinKwon for review.

@roryqi
roryqi deleted the SPARK-36223 branch November 15, 2021 08:19
@SparkQA

SparkQA commented Nov 15, 2021

Copy link
Copy Markdown

Test build #145222 has finished for PR 33510 at commit c4ff4ab.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Nov 15, 2021

Copy link
Copy Markdown

Test build #145223 has finished for PR 33510 at commit 510fa61.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwonHyukjinKwon mentioned this pull request Nov 24, 2021
Closed
HyukjinKwon
HyukjinKwon reviewed Nov 24, 2021
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/TPCDSQueryTestSuite.scala
assertResult(expectedSchema, s"Schema did not match\n$queryString") { schema }
assertResult(expectedOutput, s"Result did not match\n$queryString") { outputString }
}
val joinConfSet: Set[Map[String, String]] =

@HyukjinKwonHyukjinKwonNov 24, 2021•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why is this set? Then joinConfSet.head won't be deterministic below, and there would be no point of needSort.

@HyukjinKwonHyukjinKwonNov 24, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me fix it together at #34698

@roryqiroryqiNov 24, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, seq may be better.

HyukjinKwon added a commit that referenced this pull request Nov 25, 2021
@HyukjinKwon
[SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub Actions
27eb687
### What changes were proposed in this pull request?
This is kind of a followup for #33510 and #34641. This PR proposes to split TPC-DS build in GitHub Actions.
### Why are the changes needed?
Running these queries easily causes out-of-memory in GitHub Actions machines, and make the build flaky. We should deflake it.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
GitHub Actions in this PR should test it out.
Closes#34698 from HyukjinKwon/split-tpcds.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwonHyukjinKwonHyukjinKwon left review comments
@cloud-fancloud-fancloud-fan approved these changes
+1 more reviewer
@linhongliu-dblinhongliu-dblinhongliu-db left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@roryqi@cloud-fan@SparkQA@linhongliu-db@HyukjinKwon