[yaml] - add nullable field test for readFromBigQuery and update create core logic by derrickaw · Pull Request #35692 · apache/beam

derrickaw · 2025-07-25T15:48:51Z

This PR solves two issues that are interconnected:

fixed the beam yaml create issue by creating the full dict #35438 solved the truncated fields issue, but it also introduced the any type for a field if a row had null data. With the updated Create transform, WriteToBigQuery was complaining about TYPEINFO_NOT_SET and trying to handle a python any type ([Bug]: Yaml writeToBigQuery - java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET #35668). This PR changes the logic to where if a field has one known type and a null entry, the field will be made an optional type.
A user mentioned an issue with ReadFromBQ not handling null field entries and datetimes in some of the data and this was fixed by Another attempt to fix timestamp and decimal logical type. #35426. With this PR, we want to cover the case where there is nullable data to catch regressions better.

Note - removed some test coverage for AssignTimestamps and ValidateWithSchema transforms due to now throwing an error in Create due to this PR change. Commenting out for now. Also, ValidateWithSchema fails on any data that is Null. This should probably be fixed in a followup PR.

Tested locally:
/beam/sdks/python/apache_beam/yaml$ pytest -v integration_tests.py::BigqueryTest --test_files_dir="extended_tests/databases"

b/431833297

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2025-07-25T15:48:54Z

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

derrickaw · 2025-07-25T16:01:52Z

Run Python_Integration PreCommit 3.9

derrickaw · 2025-07-25T17:57:46Z

Run Xlang_Generated_Transforms PreCommit 3.9

github-actions · 2025-07-25T19:07:31Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

derrickaw · 2025-07-26T00:31:07Z

assign set of reviewers

github-actions · 2025-07-26T00:32:09Z

Assigning reviewers:

R: @liferoad for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

gemini-code-assist · 2025-07-27T13:27:19Z

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

liferoad · 2025-07-27T13:33:49Z

 - {user: "bob", timestamp: 7}

+ # TODO - Rethink error handling since schema mismatch in Create transform
+ # isn't allowed anymore.


If I understand this correctly, typehints.Union will be used here when they have different schemas, right?

In this particular case it will fail with types mismatch since two different types were used that were not nullable.

codecov · 2025-07-27T21:44:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.56%. Comparing base (80f1052) to head (f4629c6).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files

@@ Coverage Diff @@## master #35692 +/- ##
============================================
+ Coverage 56.53% 56.56% +0.03% 
Complexity 3381 3381 ============================================
Files 1203 1204 +1 Lines 183895 183771 -124 Branches 3490 3490 ============================================
- Hits 103964 103951 -13 + Misses 76604 76493 -111 
Partials 3327 3327

Flag	Coverage Δ
python	`80.78% <100.00%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

liferoad · 2025-07-30T14:26:35Z

+ - type: Create
+ config:
+ elements:
+ - {sdk: MapReduce, year: 2004}


@damccorm this is related to what we discussed. We already have ValidateWithSchema for YAML, which can help validate the schema. But we do not have a way to define the schema. For example, here, year could be NULL but the inferred schema from Beam is Optional[Any], which cannot be accepted by WriteToBigQuery since WriteToBigQuery does not allow any type. We should do something like ValidateWithSchema that allows users to define the output schema.

Discussed with @damccorm offline and he thought it was a good idea with adding an output schema to create and most other transforms with bad records going to error handling. I will open an issue for this (#35742). Thanks.

liferoad · 2025-07-30T14:27:01Z

+#  - type: ValidateWithSchema
+#  input: Create
+#  config:
+#  schema:


@damccorm like this.

liferoad · 2025-07-30T14:30:28Z

I recommend we close this PR for now and focus on defining the schema.

derrickaw · 2025-07-30T15:57:04Z

I recommend we close this PR for now and focus on defining the schema.

Hi @liferoad , I discussed with @damccorm offline and we were thinking that if we removed the hard check for multiple types, then this wouldn't be a breaking change anymore and would solve the issue with any type and nullability. Thoughts?

liferoad · 2025-07-30T16:16:36Z

I recommend we close this PR for now and focus on defining the schema.
Hi @liferoad , I discussed with @damccorm offline and we were thinking that if we removed the hard check for multiple types, then this wouldn't be a breaking change anymore and would solve the issue with any type and nullability. Thoughts?

Sounds good if we don't introduce a breaking change now.

derrickaw · 2025-07-30T18:09:10Z

Run Python_Examples PreCommit 3.12

derrickaw · 2025-07-30T18:18:55Z

Run Python_ML PreCommit 3.9

derrickaw · 2025-07-30T20:01:41Z

Hi @liferoad - reverted the changes to assign_timestamp and validate_with_schema.

liferoad · 2025-07-30T22:57:42Z

/gemini review

liferoad · 2025-07-30T22:58:12Z

Please check the failed workflows.

gemini-code-assist

Code Review

This pull request addresses issues related to type inference with nullable data, particularly for beam.Row objects. The changes in Create.infer_output_type to correctly handle Optional types are well-implemented. The addition of integration tests for ReadFromBigQuery with nullable data is also a valuable contribution.

derrickaw · 2025-08-01T01:21:21Z

Run Python_Coverage PreCommit

liferoad · 2025-08-01T02:06:49Z

looks like this breaks some of my new flatten tests:
https://github.com/apache/beam/pull/35692/checks?check_run_id=47167629835

derrickaw · 2025-08-01T16:04:20Z

Run Python_PVR_Flink PreCommit

derrickaw · 2025-08-01T16:09:28Z

Unit tests are flaky.

liferoad · 2025-08-01T16:13:02Z

#35764 fixes some issues. You can rebase.

github-actionsBot added python yaml labels Jul 25, 2025

derrickaw marked this pull request as ready for review July 25, 2025 19:04

derrickaw force-pushed the addBQyamlit branch from 0d53777 to cec1208Compare July 25, 2025 20:17

github-actionsBot added the Next Action: Reviewers label Jul 26, 2025

liferoad reviewed Jul 26, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/transforms/core.py Outdated

liferoad reviewed Jul 26, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/transforms/core.py

derrickaw requested a review from liferoad July 27, 2025 02:40

liferoad reviewed Jul 27, 2025
View reviewed changes

derrickaw requested a review from liferoad July 27, 2025 23:33

liferoad reviewed Jul 30, 2025
View reviewed changes

derrickaw requested a review from liferoad July 30, 2025 19:34

liferoad approved these changes Jul 30, 2025
View reviewed changes

gemini-code-assistBot reviewed Jul 30, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/transforms/core.py
Comment threadsdks/python/apache_beam/transforms/core_test.py
Comment threadsdks/python/apache_beam/yaml/extended_tests/databases/bigquery.yaml

derrickaw force-pushed the addBQyamlit branch from fce6843 to bce0bfaCompare July 31, 2025 23:26

derrickaw force-pushed the addBQyamlit branch from bce0bfa to f79b079Compare August 1, 2025 15:53

derrickaw added 15 commits August 1, 2025 16:26

add more test coverage for nullable field for BQ
6a79549

yapf changes
2a57997

change else statement to field types instead of row
a1b6862

remove error handling as not needed for now
6e75498

fix lint issues
b46336c

comment out failing create transforms
45daa2d

address comments
feb4f8e

add core create UT
6cb318c

update core logic
1696020

fix lint
d9705d8

fix brittle test
c1dac3d

update multiple type logic
e5bb876

revert changes to assign timestamps and validate
c12d82b

switch to typehints
ede7cec

updated core tests with typehints
64b99ac

derrickaw force-pushed the addBQyamlit branch from 2fe3377 to 64b99acCompare August 1, 2025 16:26

derrickaw requested a review from liferoad August 1, 2025 17:55

liferoad approved these changes Aug 1, 2025
View reviewed changes

liferoad merged commit 00c1f10 into apache:masterAug 1, 2025
90 checks passed

derrickaw deleted the addBQyamlit branch August 1, 2025 18:13

derrickaw mentioned this pull request Aug 1, 2025
[Bug]: Yaml writeToBigQuery - java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET #35668
Closed
17 tasks

Uh oh!

Conversation

derrickaw commented Jul 25, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assistBot commented Jul 25, 2025

Uh oh!

derrickaw commented Jul 25, 2025

Uh oh!

derrickaw commented Jul 25, 2025

Uh oh!

github-actionsBot commented Jul 25, 2025

Uh oh!

derrickaw commented Jul 26, 2025

Uh oh!

github-actionsBot commented Jul 26, 2025

Uh oh!

Uh oh!

Uh oh!

gemini-code-assistBot commented Jul 27, 2025

Uh oh!

liferoadJul 27, 2025

Choose a reason for hiding this comment

Uh oh!

derrickawJul 27, 2025

Choose a reason for hiding this comment

Uh oh!

codecovBot commented Jul 27, 2025

Codecov Report

Uh oh!

liferoadJul 30, 2025

Choose a reason for hiding this comment

Uh oh!

derrickawJul 30, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liferoadJul 30, 2025

Choose a reason for hiding this comment

Uh oh!

liferoad commented Jul 30, 2025

Uh oh!

derrickaw commented Jul 30, 2025

Uh oh!

liferoad commented Jul 30, 2025

Uh oh!

derrickaw commented Jul 30, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

derrickaw commented Jul 30, 2025

Uh oh!

derrickaw commented Jul 30, 2025

Uh oh!

liferoad commented Jul 30, 2025

Uh oh!

liferoad commented Jul 30, 2025

Uh oh!

gemini-code-assistBot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Aug 1, 2025

Uh oh!

liferoad commented Aug 1, 2025

Uh oh!

derrickaw commented Aug 1, 2025

Uh oh!

derrickaw commented Aug 1, 2025

Uh oh!

liferoad commented Aug 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

derrickaw commented Jul 25, 2025•
edited
Loading

derrickawJul 30, 2025•
edited
Loading

derrickaw commented Jul 30, 2025•
edited
Loading