process: add mypy-valid type annotations to samples by plamut · Pull Request #1081 · googleapis/python-bigquery

plamut · 2021-12-08T15:34:50Z

Towards #1074.

This PR adds type annotations to samples and fixes quite a few issues discovered along. It also adds a new nox session to run type checks with mypy on samples.

The mypy.ini files in samples/* subdirectories are added for the upcoming type-check sessions that will be added to the noxfiles in these directories (through the code generator).

PR checklist:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

* feat: allow cell magic body to be a $variable * Fix missing indefinitive article in error msg * Adjust test assertion to error message change * Refactor logic for extracting query variable * Explicitly warn about missing query variable name * Thest the query "variable" is not identifier case

…eapis#1073) * feat: promote `to_arrow_iterable` to public method * use correct version number * Update google/cloud/bigquery/table.py Co-authored-by: Tim Swast <swast@google.com>

* fix: apply timeout to all resumable upload requests * Fix stub in test case * Improve timeout type and other type annotations * Annnotate return type of _do_resumable_upload()

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

…taFrame (googleapis#1078) Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Towards #1076 🦕 (edit: moved to googleapis/python-db-dtypes-pandas#45 )

* revoke dataset access setup * basic template for sample * sample + test * revoke dataset access sample * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * docs: add sample for revoking dataset access - update year and string formatting * docs: add sample for revoking dataset access - move to snippets and change parameter pattern for readibility * moving update_dataset to /snippets and adjusting imports on both revoke_access and update_access * Update samples/snippets/revoke_dataset_access.py removed nested START/END tags Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/revoke_dataset_access.py update readability in API request Co-authored-by: Tim Swast <swast@google.com> * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * updated test * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * change after running test * resolving linting failure, rewriting test * removed relative import errors * remove relative mport from update_dataset_access * adding fixture to conftest.py * updated sample * updating sample to match new update_access sample * fixing region tags * consolidated tests into one file for both methods * updating test to full_dataset format * updated revoke sample * updating test * refactored sample * Update samples/snippets/conftest.py * Update samples/snippets/revoke_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/update_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/revoke_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/revoke_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * refactoring entry * added comment for entry access * Update samples/snippets/README.rst Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/dataset_access_test.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/dataset_access_test.py Co-authored-by: Tim Swast <swast@google.com> * added develper TODO in sample * add comments to samples Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Swast <swast@google.com> Co-authored-by: Peter Lamut <plamut@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com> Co-authored-by: meredithslota <meredithslota@google.com>

Samples use the code from the core library, but mypy complains about calling an untyped function in a typed context, thus some additional type annotations need to be added.

plamut · 2021-12-08T15:53:48Z

ImportError: cannot import name 'TypedDict'

This was only added to typing in Python 3.8. I actually fixed this, but not at all places, and I was only running the checks with Python 3.8, missing this. 🙃

tswast

So cool!

A few nits. Plus I'd to see some improvements to the ExternalConfig class to see if we can avoid casts in those samples.

snippet-bot · 2021-12-14T15:26:30Z

Here is the summary of changes.

You are about to add 4 region tags.

samples/snippets/revoke_dataset_access.py:20, tag bigquery_revoke_dataset_access
samples/snippets/revoke_dataset_access.py:30, tag bigquery_revoke_dataset_access
samples/snippets/update_dataset_access.py:20, tag bigquery_update_dataset_access
samples/snippets/update_dataset_access.py:49, tag bigquery_update_dataset_access

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

plamut · 2021-12-14T15:29:56Z

@tswast Since there was a new sample added to main, I merged it into this branch. Had to resolve some conflicts, thus that commit should also be checked.

The rest of the comments have been addressed, and I got rid of several cast() calls by using explicit is None assertions instead - let me know how that looks to you.

(and I didn't commit suggestions through the web UI, as that has lead to the CLA bot complaining without merit)

tswast

Awesome!

It's a bit funny to have assertions in the code samples, but not too out of place. It looks more natural than the type casts.

plamut · 2021-12-14T15:38:41Z

It's a bit funny to have assertions in the code samples, but not too out of place. It looks more natural than the type casts.

True that, but I agree that's much better than those rather ugly casts.

deps!: BigQuery Storage and pyarrow are required dependencies (#776) fix!: use nullable `Int64` and `boolean` dtypes in `to_dataframe` (#786) feat!: destination tables are no-longer removed by `create_job` (#891) feat!: In `to_dataframe`, use `dbdate` and `dbtime` dtypes from db-dtypes package for BigQuery DATE and TIME columns (#972) fix!: automatically convert out-of-bounds dates in `to_dataframe`, remove `date_as_object` argument (#972) feat!: mark the package as type-checked (#1058) feat!: default to DATETIME type when loading timezone-naive datetimes from Pandas (#1061) feat: add `api_method` parameter to `Client.query` to select `INSERT` or `QUERY` API (#967) fix: improve type annotations for mypy validation (#1081) feat: use `StandardSqlField` class for `Model.feature_columns` and `Model.label_columns` (#1117) docs: Add migration guide from version 2.x to 3.x (#1027) Release-As: 3.0.0

deps!: BigQuery Storage and pyarrow are required dependencies (googleapis#776) fix!: use nullable `Int64` and `boolean` dtypes in `to_dataframe` (googleapis#786) feat!: destination tables are no-longer removed by `create_job` (googleapis#891) feat!: In `to_dataframe`, use `dbdate` and `dbtime` dtypes from db-dtypes package for BigQuery DATE and TIME columns (googleapis#972) fix!: automatically convert out-of-bounds dates in `to_dataframe`, remove `date_as_object` argument (googleapis#972) feat!: mark the package as type-checked (googleapis#1058) feat!: default to DATETIME type when loading timezone-naive datetimes from Pandas (googleapis#1061) feat: add `api_method` parameter to `Client.query` to select `INSERT` or `QUERY` API (googleapis#967) fix: improve type annotations for mypy validation (googleapis#1081) feat: use `StandardSqlField` class for `Model.feature_columns` and `Model.label_columns` (googleapis#1117) docs: Add migration guide from version 2.x to 3.x (googleapis#1027) Release-As: 3.0.0

plamutand others added 16 commits November 16, 2021 22:57

cleanup: silence non-relevant system test warnings (googleapis#1068)
1b5dc5c

feat: promoteRowIterator.to_arrow_iterableto public method (googl…
21cd710
…eapis#1073) * feat: promote `to_arrow_iterable` to public method * use correct version number * Update google/cloud/bigquery/table.py Co-authored-by: Tim Swast <swast@google.com>

fix: apply timeout to all resumable upload requests (googleapis#1070)
3314dfb
* fix: apply timeout to all resumable upload requests * Fix stub in test case * Improve timeout type and other type annotations * Annnotate return type of _do_resumable_upload()

chore: release 2.31.0 (googleapis#1066)
a135956
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

Add type hints to magics samples
d1a9902

Add type hints to geography samples
8141548

Add type hints to snippets
0b58d9d

Add type hints to top level samples (part 1)
452f15e

Add type hints to top level samples (part 2)
ee1179a

Add type hints to top level samples (part 3)
d77d70e

Add type hints to core libraries for samples' needs
8195894
Samples use the code from the core library, but mypy complains about calling an untyped function in a typed context, thus some additional type annotations need to be added.

Add top level mypy config and samples session
6a3713b

Add mypy.ini files to samples subdirectories
c4d0ab5

plamut requested review from a team and tswast December 8, 2021 15:34

plamut requested a review from a team as a code owner December 8, 2021 15:34

plamut requested review from a team and busunkim96 December 8, 2021 15:34

product-auto-labelBot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Dec 8, 2021

plamut changed the title ~~Iss 1074~~Add mypy-valid type annotations to samplesDec 8, 2021

plamut changed the title ~~Add mypy-valid type annotations to samples~~process: add mypy-valid type annotations to samplesDec 8, 2021

Enforce late evaluation of some annotations
c0ef9a0

Install typing.TypedDict backport
86dd389

plamut force-pushed the iss-1074 branch from c00fa0b to be35bc4Compare December 8, 2021 18:32

Make sure typing.TypedDict is available in samples
278dac7

plamut force-pushed the iss-1074 branch from be35bc4 to 278dac7Compare December 8, 2021 18:34

plamut added 2 commits December 8, 2021 22:35

Use import fallback for TypedDict
2c78c72

Fix service account fixture in snippets
12e118d

product-auto-labelBot added the samples Issues that are directly related to samples. label Dec 9, 2021

chore: update python-docs-samples link to main branch (googleapis#1082)
44221f4

tswast suggested changes Dec 10, 2021
View reviewed changes

plamut added 5 commits December 14, 2021 12:02

Merge branch 'main' into iss-1074
eaaffb7

Add missing annotations to new samples
040a21c

Move the override_values default check out
fdabea3

Get rid of type cast of an optional int property
484a6ee

Simplify external config type annotations
8c40ec7

tswast approved these changes Dec 14, 2021
View reviewed changes

plamut added the automerge Merge the pull request once unit tests and other checks pass. label Dec 14, 2021

tswast merged commit 7e3721e into googleapis:v3Dec 14, 2021

gcf-merge-on-greenBot removed the automerge Merge the pull request once unit tests and other checks pass. label Dec 14, 2021

plamut deleted the iss-1074 branch December 14, 2021 16:07

tswast mentioned this pull request Dec 15, 2021
chore: sync v3 with main branch #1088
Merged

tswast mentioned this pull request Mar 29, 2022
fix!: remove out-of-date BigQuery ML protocol buffers #1178
Merged
4 tasks

release-pleaseBot mentioned this pull request Mar 29, 2022
chore(main): release 3.0.0 #1179
Merged

Navigation Menu

Uh oh!

process: add mypy-valid type annotations to samples - #1081

process: add mypy-valid type annotations to samples#1081
tswast merged 27 commits into
googleapis:v3from
plamut:iss-1074

plamut commented Dec 8, 2021

Uh oh!

plamut commented Dec 8, 2021

Uh oh!

tswast left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snippet-botBot commented Dec 14, 2021

Uh oh!

plamut commented Dec 14, 2021•
edited
Loading

Uh oh!

tswast left a comment

Uh oh!

plamut commented Dec 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

plamut commented Dec 8, 2021

Uh oh!

plamut commented Dec 8, 2021

Uh oh!

tswast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snippet-botBot commented Dec 14, 2021

Uh oh!

plamut commented Dec 14, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tswast left a comment

Choose a reason for hiding this comment

Uh oh!

plamut commented Dec 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

plamut commented Dec 14, 2021•
edited
Loading