Create Avro extension for Java SDK by aromanenko-dev · Pull Request #24294 · apache/beam

aromanenko-dev · 2022-11-21T17:50:22Z

Create a dedicated extension for Java SDK to support Avro format.

This PR mostly includes copy-pasted Avro-related code from sdks/java/core with the same package structure and it introduces almost no changes to "core" - only several access modifiers are relaxed to be able to use from other packages. It should not break any other user code.

This is the first step to use "Avro from extension" instead of "Avro from core".

Closes#24293

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2022-11-21T18:32:31Z

Codecov Report

Merging #24294 (150877e) into master (820839f) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #24294 +/- ##
==========================================
- Coverage 73.27% 73.25% -0.02% 
==========================================
Files 725 725 Lines 97444 97444 ==========================================
- Hits 71403 71385 -18 - Misses 24691 24709 +18 
Partials 1350 1350

Flag	Coverage Δ
python	`82.83% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (-2.44%)`	⬇️
...ks/python/apache_beam/runners/worker/data_plane.py	`87.57% <0.00%> (-1.70%)`	⬇️
.../python/apache_beam/transforms/periodicsequence.py	`97.01% <0.00%> (-1.50%)`	⬇️
...che_beam/runners/interactive/interactive_runner.py	`90.50% <0.00%> (-1.27%)`	⬇️
sdks/python/apache_beam/io/localfilesystem.py	`90.97% <0.00%> (-0.76%)`	⬇️
...eam/runners/portability/fn_api_runner/execution.py	`92.49% <0.00%> (-0.64%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.54% <0.00%> (-0.13%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

aromanenko-dev · 2022-12-20T16:37:09Z

R: @mosche
FYI: CC: @kennknowles @lukecwik @reuvenlax @robertwb
ptal

github-actions · 2022-12-20T17:05:08Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @Abacn for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

mosche · 2022-12-21T13:19:08Z

👀

mosche · 2022-12-21T13:26:42Z

Run Spotless PreCommit

mosche · 2022-12-21T13:29:23Z

Closes #24293

This PR doesn't close the ticket yet, right?

aromanenko-dev · 2022-12-21T15:08:36Z

@mosche

This PR doesn't close the ticket yet, right?

It does since it's just a sub-task of #24292

mosche

Some questions, pls have a look @aromanenko-dev
I also found a couple of copy-paste issues. I've checked the diff for all newly committed files. Looks fine except for the cases mentioned below.

find sdks/java -name AvroCoder.java | xargs diff --color

$ grep -r -E 'org\.apache\.beam\.sdk\.(schemas|io)(\.\w+)*\.\w*Avro' sdks/java/extensions/avro

sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java: new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: private static <T> org.apache.beam.sdk.io.AvroSource<T> createSource(
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource.@Nullable DatumReaderFactory<T> readerFactory) {
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource<?> source =
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource.from(filepattern)
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: ? (org.apache.beam.sdk.io.AvroSource<T>) source.withSchema(schema)
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: abstract org.apache.beam.sdk.io.AvroSource.@Nullable DatumReaderFactory<T>
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> factory);
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> factory) {
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: private final org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> readerFactory;
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> readerFactory) {
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java: org.apache.beam.sdk.io.AvroSource.from(getFilepattern())
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java: * A specialization of {@link DynamicDestinations} for {@link org.apache.beam.sdk.io.AvroIO}. In
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java: * Return a {@link org.apache.beam.sdk.io.AvroSink.DatumWriterFactory} for a given destination. If

aromanenko-dev · 2022-12-26T12:45:55Z

Run Java_GCP_IO_Direct PreCommit

aromanenko-dev · 2022-12-26T14:04:58Z

Run Java_Pulsar_IO_Direct PreCommit

aromanenko-dev · 2022-12-26T15:30:02Z

@mosche Thanks for review, I believe I addressed all your comments, PTAL

mosche · 2022-12-28T09:05:03Z

👀

mosche

Thanks @aromanenko-dev, looking good now 🎄 ✨
Feel free to merge!

github-actionsBot added build extensions java labels Nov 21, 2022

aromanenko-dev changed the title ~~Create an Avro extension for Java SDK~~[WIP] Create an Avro extension for Java SDKNov 21, 2022

aromanenko-dev marked this pull request as draft November 21, 2022 17:51

aromanenko-dev force-pushed the extensions_avro branch 2 times, most recently from 6fc5e88 to 67240a7Compare December 20, 2022 16:26

aromanenko-dev marked this pull request as ready for review December 20, 2022 16:28

aromanenko-dev changed the title ~~[WIP] Create an Avro extension for Java SDK~~Create Avro extension for Java SDKDec 20, 2022

github-actionsBot added the Next Action: Reviewers label Dec 20, 2022

mosche reviewed Dec 21, 2022
View reviewed changes

Comment threadsdks/java/extensions/avro/build.gradle Outdated

mosche requested changes Dec 21, 2022
View reviewed changes

aromanenko-dev force-pushed the extensions_avro branch from 67240a7 to 06dd451Compare December 21, 2022 15:33

mosche reviewed Dec 22, 2022
View reviewed changes

Comment thread...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java Outdated

mosche reviewed Dec 22, 2022
View reviewed changes

Comment thread...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java Outdated

aromanenko-dev force-pushed the extensions_avro branch from 06dd451 to 8ec7419Compare December 22, 2022 17:05

aromanenko-dev added 3 commits December 26, 2022 13:47

[apache#24292] Create Avro extension for Java SDK
873e0d0

Address the review comments
8b532db

Applyapache#24454changes
150877e

aromanenko-dev force-pushed the extensions_avro branch from 8ec7419 to 150877eCompare December 26, 2022 13:44

mosche self-requested a review December 28, 2022 09:48

mosche approved these changes Dec 28, 2022
View reviewed changes

aromanenko-dev merged commit bab773f into apache:masterDec 28, 2022

aromanenko-dev deleted the extensions_avro branch December 28, 2022 14:53

Uh oh!

Conversation

aromanenko-dev commented Nov 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

codecovBot commented Nov 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aromanenko-dev commented Dec 20, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Dec 20, 2022

Uh oh!

mosche commented Dec 21, 2022

Uh oh!

mosche commented Dec 21, 2022

Uh oh!

mosche commented Dec 21, 2022

Uh oh!

Uh oh!

aromanenko-dev commented Dec 21, 2022

Uh oh!

mosche left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aromanenko-dev commented Dec 26, 2022

Uh oh!

aromanenko-dev commented Dec 26, 2022

Uh oh!

aromanenko-dev commented Dec 26, 2022

Uh oh!

mosche commented Dec 28, 2022

Uh oh!

mosche left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aromanenko-dev commented Nov 21, 2022•
edited
Loading

codecovBot commented Nov 21, 2022•
edited
Loading

aromanenko-dev commented Dec 20, 2022•
edited
Loading

mosche left a comment •
edited
Loading