Skip to content
This repository was archived by the owner on Mar 6, 2026. It is now read-only.

fix: add warning when encountering unknown field types - #1989

Merged
Linchin merged 15 commits into
googleapis:mainfrom
suzmue:warnConversion
Aug 13, 2024
Merged

fix: add warning when encountering unknown field types #1989
Linchin merged 15 commits into
googleapis:mainfrom
suzmue:warnConversion

Conversation

@suzmue

Copy link
Copy Markdown
Contributor

Fixes#1988

@suzmue
suzmue requested review from a team and tswastAugust 5, 2024 21:14
@product-auto-labelproduct-auto-labelBot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery API. labels Aug 5, 2024
Comment threadgoogle/cloud/bigquery/_helpers.py Outdated
converter = _CELLDATA_FROM_JSON.get(field.field_type, lambda value, _: value)
def default_converter(value, field):
warnings.warn(
"Unknown type '{}' for field '{}'.".format(field.field_type, field.name),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be explicit and say "Behavior reading this type is not officially supported and may change in future." or something scary along those lines.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we mark this as "fixed", I think the inverse direction is needed too (where we write JSON from a record), which is used in methods like client.insert_rows() (though maybe that already fails when encountering unknown types?)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Yeah looks like

# Unknown fields should not be silently dropped, include them. Since there
and
converter=_SCALAR_VALUE_TO_JSON_ROW.get(field.field_type)
are two points. The first there is no type info so probably just _scalar_field_to_json needs to be updated. Updated PR to include a warning for the write.

add warning for write and warn about future behavior changes

def _field_from_json(resource, field):
converter = _CELLDATA_FROM_JSON.get(field.field_type, lambda value, _: value)
def default_converter(value, field):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can move this function to outside, so we can reuse it for _scalar_field_to_json() as well?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up just moving the warning itself to a new function, since the converter in _scalar_field_to_json has a different function signature.

@Linchin
Linchin self-requested a review August 12, 2024 22:21
@LinchinLinchin added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 13, 2024
@yoshi-kokoroyoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 13, 2024
@Linchin

Copy link
Copy Markdown
Contributor

I think the docs test isn't failing due to changes in this PR, it appears to relate to the docker image used. I will approve this PR and force merge.

LGTM.

@Linchin
Linchin merged commit 8f5a41d into googleapis:mainAug 13, 2024
@release-pleaserelease-pleaseBot mentioned this pull request Aug 13, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

api: bigqueryIssues related to the googleapis/python-bigquery API.size: mPull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a warning in the type mapping if no conversion function is found

4 participants

@suzmue@Linchin@tswast@yoshi-kokoro