Skip to content

BigQuery Storage: Add more in-depth system tests - #8992

Merged
plamut merged 14 commits into
googleapis:masterfrom
plamut:iss-8983
Aug 21, 2019
Merged

BigQuery Storage: Add more in-depth system tests #8992
plamut merged 14 commits into
googleapis:masterfrom
plamut:iss-8983

Conversation

@plamut

@plamutplamut commented Aug 8, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

Closes#8983.

This PR adds additional system tests for BigQuery Storage. It does not include the last item from the issue description, i.e. the long-running test, at least not until decided otherwise.

Things to mind

  • One of the tests depends on the fix in BigQuery: Fix schema recognition of struct field types #9001 to pass. Tests need to be re-run once that is merged.
  • (see the comment below) There is currently an inconsistent type returned for datetime columns (string instead of a datetime instance), affecting the same test.
    Created #9066 for that.

@plamutplamut added the api: bigquerystorage Issues related to the BigQuery Storage API. label Aug 8, 2019
@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 8, 2019
plamut
plamut commented Aug 9, 2019
View reviewed changes
Comment threadbigquery_storage/tests/system/test_reader.py
# Compare datetime separately, AVRO and PYARROW return different object types,
# although they should both represent the same value.
expected_pattern = re.compile(r"2005-10-26( |T)19:49:41")
assert expected_pattern.match(str(rows[0]["datetime_field"]))

@plamutplamutAug 9, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sanity check - is this expected, or should both AVRO and PYARROW return the exact same type?

Right now one returns a plain ISO datetime string, while the other returns a Timestamp object..

@plamutplamutAug 9, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answer:
"Expected" is a datetime.datetime object with no timezone. Actual is a string. That's an open issue.

@plamut

plamut commented Aug 9, 2019

Copy link
Copy Markdown
ContributorAuthor

FWIW, a system test failure with STRUCT records will be fixed in #9001.

@plamut
plamutforce-pushed the iss-8983 branch 2 times, most recently from e7b1ed2 to c5f5867CompareAugust 12, 2019 12:59
@plamut
plamut marked this pull request as ready for review August 12, 2019 15:34
@plamut
plamut requested a review from a teamAugust 12, 2019 15:34
@tseaver

tseaver commented Aug 12, 2019

Copy link
Copy Markdown
Contributor

@plamut System tests are failing in setup with the error, Field person_struct_field is type RECORD but has no schema.

@plamut

plamut commented Aug 12, 2019

Copy link
Copy Markdown
ContributorAuthor

@tseaver Indeed, that's the bug that will be fixed with #9001

tswast
tswast reviewed Aug 14, 2019
View reviewed changes

@tswasttswast left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! A few questions.

Comment threadbigquery_storage/noxfile.py Outdated
Comment threadbigquery_storage/tests/system/conftest.py Outdated
Comment threadbigquery_storage/tests/system/test_reader.py Outdated
@plamutplamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 14, 2019
@yoshi-kokoroyoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 14, 2019
@plamutplamut added the needs work This is a pull request that needs a little love. label Aug 14, 2019
@plamutplamut mentioned this pull request Aug 15, 2019
Closed
@plamut
Add TODO comment to replace a test helper method
6da135e
A similar method is planned to be added to the library itself, and when
done, the _add_rows() will not be needed anymore.
@plamut
Extract BQ client to session fixture in tests
222ab1a
Creating a client once per system tests session avoids the overhead
of authenticating before each test case.
@plamut
Only create BQ storage client once per test run
b373e3f
Creating a client just once avoids the auth overhead on every system
test case.
@plamut
plamut requested a review from tswastAugust 15, 2019 11:06
@plamutplamut removed the needs work This is a pull request that needs a little love. label Aug 15, 2019
@tswast

tswast commented Aug 21, 2019

Copy link
Copy Markdown
Contributor

I've filed https://github.com/googleapis/google-cloud-python/issues/9066 for the Avro DATETIME issue.

@plamut
plamut merged commit dec9f69 into googleapis:masterAug 21, 2019
@plamut
plamut deleted the iss-8983 branch August 21, 2019 07:42
@tswasttswast mentioned this pull request Aug 21, 2019
Merged
@plamutplamut mentioned this pull request Aug 23, 2019
Merged
HemangChothani pushed a commit to HemangChothani/google-cloud-python that referenced this pull request Aug 29, 2019
@plamut
BigQuery Storage: Add more in-depth system tests (googleapis#8992)
42150d0
* Add additional BQ storage system test fixtures
* Add reader column selection system test
* Add basic reader system test
* Add reader with row filter system test
* Add reading data with snapshot system test
* Add reading column partitioned table system test
* Add system test for column types data conversions
* Add ingestion time partitioned table system test
* Add system test for resuming a read at an offset
* Remove unnecessary protobuf install in noxfile
* Add TODO comment to replace a test helper method
A similar method is planned to be added to the library itself, and when
done, the _add_rows() will not be needed anymore.
* Extract BQ client to session fixture in tests
Creating a client once per system tests session avoids the overhead
of authenticating before each test case.
* Only create BQ storage client once per test run
Creating a client just once avoids the auth overhead on every system
test case.
* Add common credentials fixture for system tests
emar-kar pushed a commit to MaxxleLLC/google-cloud-python that referenced this pull request Sep 18, 2019
@plamut@emar-kar
BigQuery Storage: Add more in-depth system tests (googleapis#8992)
aad16d2
* Add additional BQ storage system test fixtures
* Add reader column selection system test
* Add basic reader system test
* Add reader with row filter system test
* Add reading data with snapshot system test
* Add reading column partitioned table system test
* Add system test for column types data conversions
* Add ingestion time partitioned table system test
* Add system test for resuming a read at an offset
* Remove unnecessary protobuf install in noxfile
* Add TODO comment to replace a test helper method
A similar method is planned to be added to the library itself, and when
done, the _add_rows() will not be needed anymore.
* Extract BQ client to session fixture in tests
Creating a client once per system tests session avoids the overhead
of authenticating before each test case.
* Only create BQ storage client once per test run
Creating a client just once avoids the auth overhead on every system
test case.
* Add common credentials fixture for system tests
parthea pushed a commit that referenced this pull request Aug 21, 2025
@plamut
BigQuery Storage: Add more in-depth system tests (#8992)
4563b90
* Add additional BQ storage system test fixtures
* Add reader column selection system test
* Add basic reader system test
* Add reader with row filter system test
* Add reading data with snapshot system test
* Add reading column partitioned table system test
* Add system test for column types data conversions
* Add ingestion time partitioned table system test
* Add system test for resuming a read at an offset
* Remove unnecessary protobuf install in noxfile
* Add TODO comment to replace a test helper method
A similar method is planned to be added to the library itself, and when
done, the _add_rows() will not be needed anymore.
* Extract BQ client to session fixture in tests
Creating a client once per system tests session avoids the overhead
of authenticating before each test case.
* Only create BQ storage client once per test run
Creating a client just once avoids the auth overhead on every system
test case.
* Add common credentials fixture for system tests
parthea pushed a commit that referenced this pull request Sep 16, 2025
@plamut@parthea
BigQuery Storage: Add more in-depth system tests (#8992)
0e2bd47
* Add additional BQ storage system test fixtures
* Add reader column selection system test
* Add basic reader system test
* Add reader with row filter system test
* Add reading data with snapshot system test
* Add reading column partitioned table system test
* Add system test for column types data conversions
* Add ingestion time partitioned table system test
* Add system test for resuming a read at an offset
* Remove unnecessary protobuf install in noxfile
* Add TODO comment to replace a test helper method
A similar method is planned to be added to the library itself, and when
done, the _add_rows() will not be needed anymore.
* Extract BQ client to session fixture in tests
Creating a client once per system tests session avoids the overhead
of authenticating before each test case.
* Only create BQ storage client once per test run
Creating a client just once avoids the auth overhead on every system
test case.
* Add common credentials fixture for system tests
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tswasttswasttswast approved these changes

Assignees

No one assigned

Labels

api: bigquerystorageIssues related to the BigQuery Storage API.cla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

BigQuery Storage: Add more in-depth system tests covering all data formats and field data types

5 participants

@plamut@tseaver@tswast@googlebot@yoshi-kokoro