Skip to content

[SPARK-31905][SS] Add compatibility tests for streaming state store format - #28725

Closed
xuanyuanking wants to merge 10 commits into
apache:masterfrom
xuanyuanking:compatibility_check
Closed

[SPARK-31905][SS] Add compatibility tests for streaming state store format#28725
xuanyuanking wants to merge 10 commits into
apache:masterfrom
xuanyuanking:compatibility_check

Conversation

@xuanyuanking

@xuanyuankingxuanyuanking commented Jun 4, 2020

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Add compatibility tests for streaming state store format.

Why are the changes needed?

After SPARK-31894, we have a validation checking for the streaming state store. It's better to add integrated tests in the PR builder as soon as the breaking changes introduced.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Test only.

@SparkQA

Copy link
Copy Markdown

Test build #123524 has finished for PR 28725 at commit b6b7b18.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@xuanyuankingxuanyuanking changed the title [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format[WIP][SPARK-31905][SS] Add compatibility tests for streaming state store formatJun 15, 2020
@SparkQA

Copy link
Copy Markdown

Test build #124127 has finished for PR 28725 at commit f311e82.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class StreamingStateStoreFormatCompatibilitySuite extends StreamTest

)
}

test("deduplicate with all columns") {

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @maropu@dongjoon-hyun
This new integrated test will return the wrong answer without #28830.

@xuanyuanking

Copy link
Copy Markdown
MemberAuthor

#28707 has been merged, these integration tests are ready for review.
cc @cloud-fan@HeartSaVioR for taking a look when you have time.

@xuanyuankingxuanyuanking changed the title [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format[SPARK-31905][SS] Add compatibility tests for streaming state store formatJun 19, 2020
@SparkQA

Copy link
Copy Markdown

Test build #124276 has finished for PR 28725 at commit 9ad2c06.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@xuanyuanking

Copy link
Copy Markdown
MemberAuthor

Thanks for the review!

@xuanyuanking
xuanyuanking deleted the compatibility_check branch June 22, 2020 08:27
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@xuanyuanking@SparkQA@cloud-fan