Skip to content

[SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub Actions - #34698

Closed
HyukjinKwon wants to merge 2 commits into
apache:masterfrom
HyukjinKwon:split-tpcds
Closed

[SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub Actions#34698
HyukjinKwon wants to merge 2 commits into
apache:masterfrom
HyukjinKwon:split-tpcds

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This is kind of a followup for #33510 and #34641. This PR proposes to split TPC-DS build in GitHub Actions.

Why are the changes needed?

Running these queries easily causes out-of-memory in GitHub Actions machines, and make the build flaky. We should deflake it.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

GitHub Actions in this PR should test it out.

@HyukjinKwon
HyukjinKwon marked this pull request as draft November 24, 2021 10:47
@HyukjinKwonHyukjinKwon changed the title [SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub Actions[WIP][SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub ActionsNov 24, 2021
@SparkQA

This comment has been minimized.

@HyukjinKwonHyukjinKwon changed the title [WIP][SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub Actions[SPARK-37453][INFRA][SQL][TESTS] Split TPC-DS build in GitHub ActionsNov 24, 2021
@HyukjinKwon
HyukjinKwon marked this pull request as ready for review November 24, 2021 12:17
@SparkQA

This comment has been minimized.

@HyukjinKwon
HyukjinKwonforce-pushed the split-tpcds branch 3 times, most recently from 2ffa2da to 36b3473CompareNovember 24, 2021 13:00

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I am here, I ended up with refactoring this code part here. No behavior change. Everything is as was.

@SparkQA

This comment has been minimized.

@HyukjinKwon

Copy link
Copy Markdown
MemberAuthor

cc @cloud-fan, @linhongliu-db and @jerqi

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

Comment thread.github/workflows/build_and_test.yml Outdated
@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@SparkQA

This comment has been minimized.

@HyukjinKwon

Copy link
Copy Markdown
MemberAuthor

Tests passed.

Merged to master.

@SparkQA

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50068/

@SparkQA

Copy link
Copy Markdown

Test build #145596 has finished for PR 34698 at commit 8fa16bb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
HyukjinKwon deleted the split-tpcds branch January 4, 2022 00:51
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@HyukjinKwon@SparkQA@cloud-fan@roryqi