Skip to content

[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record count - #23674

Closed
sumitsu wants to merge 4 commits into
apache:masterfrom
sumitsu:json_emptyline_count_test
Closed

[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record count#23674
sumitsu wants to merge 4 commits into
apache:masterfrom
sumitsu:json_emptyline_count_test

Conversation

@sumitsu

@sumitsusumitsu commented Jan 28, 2019

Copy link
Copy Markdown

What changes were proposed in this pull request?

This PR consists of the test components of #23665 only, minus the associated patch from that PR.

It adds a new unit test to JsonSuite which verifies that the count() returned from a DataFrame loaded from JSON containing empty lines does not include those empty lines in the record count. The test runs count prior to otherwise reading data from the DataFrame, so as to catch future cases where a pre-parsing optimization might result in count results inconsistent with existing behavior.

This PR is intended to be deployed alongside #23667; master currently causes the test to fail, as described in SPARK-26745.

How was this patch tested?

Manual testing, existing JsonSuite unit tests.

@sumitsusumitsu mentioned this pull request Jan 28, 2019
Closed
@HyukjinKwon

HyukjinKwon commented Jan 31, 2019

Copy link
Copy Markdown
Member

ok to test

HyukjinKwon
HyukjinKwon reviewed Jan 31, 2019
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala Outdated
HyukjinKwon
HyukjinKwon reviewed Jan 31, 2019
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala Outdated
HyukjinKwon
HyukjinKwon reviewed Jan 31, 2019
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala Outdated
@HyukjinKwonHyukjinKwon changed the title [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record countJan 31, 2019
@SparkQA

SparkQA commented Jan 31, 2019

Copy link
Copy Markdown

Test build #101935 has finished for PR 23674 at commit 739bbf0.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sumitsu

sumitsu commented Feb 6, 2019

Copy link
Copy Markdown
Author

@HyukjinKwon updated to incorporate your review comments: 63ab5bd (sorry about the delay)

@SparkQA

SparkQA commented Feb 6, 2019

Copy link
Copy Markdown

Test build #102057 has finished for PR 23674 at commit 63ab5bd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Feb 6, 2019

Copy link
Copy Markdown
Member

Merged to master.

@asfgitasfgit closed this in 63bced9Feb 6, 2019
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
Branden SmithJackey Lee
[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record…
db32fc0
… count
## What changes were proposed in this pull request?
This PR consists of the `test` components of apache#23665 only, minus the associated patch from that PR.
It adds a new unit test to `JsonSuite` which verifies that the `count()` returned from a `DataFrame` loaded from JSON containing empty lines does not include those empty lines in the record count. The test runs `count` prior to otherwise reading data from the `DataFrame`, so as to catch future cases where a pre-parsing optimization might result in `count` results inconsistent with existing behavior.
This PR is intended to be deployed alongside apache#23667; `master` currently causes the test to fail, as described in [SPARK-26745](https://issues.apache.org/jira/browse/SPARK-26745).
## How was this patch tested?
Manual testing, existing `JsonSuite` unit tests.
Closesapache#23674 from sumitsu/json_emptyline_count_test.
Authored-by: Branden Smith <branden.smith@publicismedia.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
dongjoon-hyun pushed a commit that referenced this pull request Apr 27, 2019
Branden Smith@dongjoon-hyun
[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record…
ba9e12d
… count
This PR consists of the `test` components of #23665 only, minus the associated patch from that PR.
It adds a new unit test to `JsonSuite` which verifies that the `count()` returned from a `DataFrame` loaded from JSON containing empty lines does not include those empty lines in the record count. The test runs `count` prior to otherwise reading data from the `DataFrame`, so as to catch future cases where a pre-parsing optimization might result in `count` results inconsistent with existing behavior.
This PR is intended to be deployed alongside #23667; `master` currently causes the test to fail, as described in [SPARK-26745](https://issues.apache.org/jira/browse/SPARK-26745).
Manual testing, existing `JsonSuite` unit tests.
Closes#23674 from sumitsu/json_emptyline_count_test.
Authored-by: Branden Smith <branden.smith@publicismedia.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 63bced9)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@dongjoon-hyun

dongjoon-hyun commented Apr 27, 2019

Copy link
Copy Markdown
Member

Hi, @sumitsu and @HyukjinKwon . I backported this to branch-2.4 since Spark 2.4.x will be the last version of 2.x lines.

@HyukjinKwon

HyukjinKwon commented Apr 28, 2019

Copy link
Copy Markdown
Member

okie

kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 23, 2019
Branden Smith@kai-chi
[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record…
792113b
… count
This PR consists of the `test` components of apache#23665 only, minus the associated patch from that PR.
It adds a new unit test to `JsonSuite` which verifies that the `count()` returned from a `DataFrame` loaded from JSON containing empty lines does not include those empty lines in the record count. The test runs `count` prior to otherwise reading data from the `DataFrame`, so as to catch future cases where a pre-parsing optimization might result in `count` results inconsistent with existing behavior.
This PR is intended to be deployed alongside apache#23667; `master` currently causes the test to fail, as described in [SPARK-26745](https://issues.apache.org/jira/browse/SPARK-26745).
Manual testing, existing `JsonSuite` unit tests.
Closesapache#23674 from sumitsu/json_emptyline_count_test.
Authored-by: Branden Smith <branden.smith@publicismedia.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 63bced9)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 25, 2019
Branden Smith@kai-chi
[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record…
371ad36
… count
This PR consists of the `test` components of apache#23665 only, minus the associated patch from that PR.
It adds a new unit test to `JsonSuite` which verifies that the `count()` returned from a `DataFrame` loaded from JSON containing empty lines does not include those empty lines in the record count. The test runs `count` prior to otherwise reading data from the `DataFrame`, so as to catch future cases where a pre-parsing optimization might result in `count` results inconsistent with existing behavior.
This PR is intended to be deployed alongside apache#23667; `master` currently causes the test to fail, as described in [SPARK-26745](https://issues.apache.org/jira/browse/SPARK-26745).
Manual testing, existing `JsonSuite` unit tests.
Closesapache#23674 from sumitsu/json_emptyline_count_test.
Authored-by: Branden Smith <branden.smith@publicismedia.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 63bced9)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Aug 1, 2019
Branden Smith@kai-chi
[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record…
bb2dc4e
… count
This PR consists of the `test` components of apache#23665 only, minus the associated patch from that PR.
It adds a new unit test to `JsonSuite` which verifies that the `count()` returned from a `DataFrame` loaded from JSON containing empty lines does not include those empty lines in the record count. The test runs `count` prior to otherwise reading data from the `DataFrame`, so as to catch future cases where a pre-parsing optimization might result in `count` results inconsistent with existing behavior.
This PR is intended to be deployed alongside apache#23667; `master` currently causes the test to fail, as described in [SPARK-26745](https://issues.apache.org/jira/browse/SPARK-26745).
Manual testing, existing `JsonSuite` unit tests.
Closesapache#23674 from sumitsu/json_emptyline_count_test.
Authored-by: Branden Smith <branden.smith@publicismedia.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 63bced9)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwonHyukjinKwonHyukjinKwon approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sumitsu@HyukjinKwon@SparkQA@dongjoon-hyun