Skip to content

[SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders - #25792

Closed
HeartSaVioR wants to merge 2 commits into
apache:masterfrom
HeartSaVioR:SPARK-23539-FOLLOWUP
Closed

[SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders#25792
HeartSaVioR wants to merge 2 commits into
apache:masterfrom
HeartSaVioR:SPARK-23539-FOLLOWUP

Conversation

@HeartSaVioR

@HeartSaVioRHeartSaVioR commented Sep 15, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This patch adds new UT to ensure existing query (before Spark 3.0.0) with checkpoint doesn't break with default configuration of "includeHeaders" being introduced via SPARK-23539.

This patch also modifies existing test which checks type of columns to also check headers column as well.

Why are the changes needed?

The patch adds missing tests which guarantees backward compatibility of the change of SPARK-23539.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT passed.

@HeartSaVioR

HeartSaVioR commented Sep 15, 2019

Copy link
Copy Markdown
ContributorAuthor

This patch is to address my own comment which PR was merged without addressing this.
#22282 (comment)

@HeartSaVioRHeartSaVioR changed the title [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders[SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeadersSep 15, 2019
@SparkQA

SparkQA commented Sep 15, 2019

Copy link
Copy Markdown

Test build #110604 has finished for PR 25792 at commit a3b5824.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HeartSaVioR

HeartSaVioR commented Sep 15, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Actually the other intention was let test fail when "includeHeaders" is set to "true", as we suspected from #22282 (comment), but turns out it doesn't break even the option is set to true. It's because of the mechanism how dropDuplicate works - row is stored as key, and getting value via key doesn't leverage type while comparing as both hashCode and equals of row don't use type.

Luckily it doesn't let query crash, but it would be ideal to do type check and avoid the possibility of crashing in runtime - SPARK-27237 (#24173) has been proposed to address this.

@HeartSaVioR

HeartSaVioR commented Sep 15, 2019

Copy link
Copy Markdown
ContributorAuthor

cc. @zsxwing@srowen as reviewers with commit privileges of #24173 , and @dongjinleekr as author of #24173

HyukjinKwon
HyukjinKwon approved these changes Sep 16, 2019
View reviewed changes

@HyukjinKwonHyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems fine to me.

dongjinleekr
dongjinleekr reviewed Sep 16, 2019
View reviewed changes
Comment thread...kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
intercept[IllegalArgumentException] { test(minPartitions = "-1", 1, true) }
}

test("default config of includeHeader doesn't break existing query from Spark 2.4") {

@dongjinleekrdongjinleekrSep 16, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for this test. 😄

@SparkQA

SparkQA commented Sep 16, 2019

Copy link
Copy Markdown

Test build #4868 has finished for PR 25792 at commit a3b5824.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HeartSaVioR

HeartSaVioR commented Sep 16, 2019

Copy link
Copy Markdown
ContributorAuthor

FYI, CI build just got passed.

@srowen

srowen commented Sep 16, 2019

Copy link
Copy Markdown
Member

Merged to master

@HeartSaVioR

HeartSaVioR commented Sep 16, 2019

Copy link
Copy Markdown
ContributorAuthor

Thanks all for reviewing and merging!

@HeartSaVioR
HeartSaVioR deleted the SPARK-23539-FOLLOWUP branch September 16, 2019 20:38
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowensrowensrowen approved these changes
@HyukjinKwonHyukjinKwonHyukjinKwon approved these changes
+1 more reviewer
@dongjinleekrdongjinleekrdongjinleekr left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

STRUCTURED STREAMINGTESTS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@HeartSaVioR@SparkQA@srowen@dongjinleekr@HyukjinKwon@dongjoon-hyun