Add GeneratedClassRowTypeConstraint by TheNeuralBit · Pull Request #22679 · apache/beam

TheNeuralBit · 2022-08-10T23:11:35Z

A RowTypeConstraint that wraps a generated NamedTuple type cannot be pickled, because the generated type cannot be pickled. This PR adds a specialization, GeneratedClassRowTypeConstraint, with a custom __reduce__ that avoids pickling the user type.

Extracted from #22575

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…nstraint

codecov · 2022-08-10T23:32:22Z

Codecov Report

Merging #22679 (59686fe) into master (c9c57a7) will decrease coverage by 0.00%.
The diff coverage is 97.29%.

@@ Coverage Diff @@## master #22679 +/- ##
==========================================
- Coverage 74.19% 74.19% -0.01% 
==========================================
Files 708 709 +1 Lines 93465 93498 +33 ==========================================
+ Hits 69347 69367 +20 - Misses 22843 22856 +13 
Partials 1275 1275

Flag	Coverage Δ
python	`83.58% <97.29%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ks/python/apache_beam/typehints/schema_registry.py	`95.23% <95.23%> (ø)`
sdks/python/apache_beam/typehints/schemas.py	`94.17% <97.14%> (-0.15%)`	⬇️
sdks/python/apache_beam/coders/row_coder.py	`94.54% <100.00%> (+0.05%)`	⬆️
sdks/python/apache_beam/transforms/core.py	`92.94% <100.00%> (ø)`
sdks/python/apache_beam/typehints/row_type.py	`97.01% <100.00%> (+0.46%)`	⬆️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (-2.44%)`	⬇️
sdks/python/apache_beam/utils/subprocess_server.py	`56.54% <0.00%> (-2.20%)`	⬇️
...python/apache_beam/runners/worker/worker_status.py	`77.53% <0.00%> (-2.18%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`88.62% <0.00%> (-0.32%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`91.71% <0.00%> (-0.31%)`	⬇️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2022-08-11T01:34:16Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

TheNeuralBit · 2022-08-11T15:51:03Z

R: @yeandy

Since valentyn is out

github-actions · 2022-08-11T15:54:12Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

yeandy

LGTM. Probably need to run formatter. Also seeing some Dataframe schema test errors, but didn't look too much into that.

yeandy · 2022-08-11T17:49:36Z

+ schema_options: Optional[Sequence[Tuple[str, Any]]] = None,
+ field_options: Optional[Dict[str, Sequence[Tuple[str, Any]]]] = None,
+ schema_registry: SchemaTypeRegistry = None,
+ ) -> RowTypeConstraint:


Suggested change
) ->RowTypeConstraint:
) ->GeneratedClassRowTypeConstraint:

I actually prefer to use the base-class here. GeneratedClassRowTypeConstraint can be an implementation detail.

yeandy · 2022-08-11T21:52:50Z

+ row_proto = schema_pb2.FieldType(
+ row_type=schema_pb2.RowType(


Why does schema_pb2.RowType have to be wrapped by schema_pb2.FieldType?

Looked again, looks like top level of protos are schema_pb2.FieldType I believe

yeandy · 2022-08-11T22:13:50Z

+ self._fields,
+ self._schema_id,
+ self._schema_options,
+ self._field_options))


Nice. Do you it would be clear to explicitly add None for the 5th item (schema_registry)? I was initially looking for the 5 args for from_fields, but only saw 4 😄 . I'm perfectly ok with the current implementation though.

Good idea, thanks

yeandy · 2022-08-11T22:17:35Z

+class GeneratedClassRowTypeConstraint(RowTypeConstraint):
+ """Specialization of RowTypeConstraint which relies on a generated user_type.
+
+ Since the generated user_type cannot be pickled, we supply a custom __reduce__


the generated user_type cannot be pickled
Can you please explain why this is the case?

I actually don't fully understand it, but it's been a consistent issue with the Schema code. Each pickle library (built-in, dill, cloudpickle) fails for a different reason. I filed #22714 to track this, and added a (skipped) test.

Co-authored-by: Andy Ye <andyye333@gmail.com>

chamikaramj · 2022-08-16T21:31:11Z

Probably this resulted in x-lang test suite failures here: #22748

…duced in apache#22679

* Add (failing) pickling tests * Add GeneratedClassRowTypeConstraint, plumb options * Add top-level option conversion functions * Refactor NamedTuple generation, always create GeneratedClassRowTypeConstraint * Move registry to apache_beam.typehints.schema_registry * yapf,lint * fixup! Move registry to apache_beam.typehints.schema_registry * Apply suggestions from code review Co-authored-by: Andy Ye <andyye333@gmail.com> * Add None SchemaRegistry * Add skipped test for pickling generated type Co-authored-by: Andy Ye <andyye333@gmail.com>

TheNeuralBit added 4 commits August 10, 2022 16:06

Add (failing) pickling tests
71992e7

Add GeneratedClassRowTypeConstraint, plumb options
b1e49ae

Add top-level option conversion functions
6ebbbf6

Refactor NamedTuple generation, always create GeneratedClassRowTypeCo…
bb17e7a
…nstraint

github-actionsBot added the python label Aug 10, 2022

TheNeuralBit mentioned this pull request Aug 10, 2022
Add BatchConverter implementations for pandas types, use Batched DoFns in DataFrame convert utilities #22575
Merged

TheNeuralBit added 3 commits August 10, 2022 16:49

Move registry to apache_beam.typehints.schema_registry
d4e7cfb

yapf,lint
8529cb4

fixup! Move registry to apache_beam.typehints.schema_registry
f60fbab

github-actionsBot added the Next Action: Reviewers label Aug 11, 2022

yeandy approved these changes Aug 11, 2022
View reviewed changes

TheNeuralBitand others added 2 commits August 12, 2022 09:27

Apply suggestions from code review
0f20d6f
Co-authored-by: Andy Ye <andyye333@gmail.com>

Add None SchemaRegistry
2ee9040

TheNeuralBit mentioned this pull request Aug 12, 2022
[Bug]: Python schema generated types cannot be pickled #22714
Open

Add skipped test for pickling generated type
59686fe

TheNeuralBit force-pushed the generated-class-rowtypeconstraint branch from e3e71cc to 59686feCompare August 12, 2022 17:02

TheNeuralBit merged commit 3173b50 into apache:masterAug 12, 2022

chamikaramj mentioned this pull request Aug 16, 2022
[Bug]: ValidateRunnerXlangTest$SingleInputOutputTest.test failing for various x-lang test suites #22748
Open

TheNeuralBit mentioned this pull request Aug 25, 2022
[Bug]: Type inference failing for Python SDK with External transforms and beam.Row #22854
Closed

Abacn mentioned this pull request Aug 26, 2022
Support Timestamp type in xlang JDBC Read and Write #22561
Merged
4 tasks

Abacn added a commit to Abacn/beam that referenced this pull request Aug 26, 2022
Fix logical type with same language type gets completely hidden intro…
b0484e7
…duced in apache#22679

Abacn mentioned this pull request Aug 25, 2023
[Bug]: Reading from BigQuery provides inconsistent schemas #28151
Closed
15 tasks

		row_proto = schema_pb2.FieldType(
		row_type=schema_pb2.RowType(

Uh oh!

Conversation

TheNeuralBit commented Aug 10, 2022

GitHub Actions Tests Status (on master branch)

Uh oh!

codecovBot commented Aug 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actionsBot commented Aug 11, 2022

Uh oh!

TheNeuralBit commented Aug 11, 2022

Uh oh!

github-actionsBot commented Aug 11, 2022

Uh oh!

yeandy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chamikaramj commented Aug 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecovBot commented Aug 10, 2022•
edited
Loading