Skip to content

Add standard {ReadFrom,WriteTo}{Csv,Json} yaml endpoints. - #28268

Merged
robertwb merged 10 commits into
apache:masterfrom
robertwb:yaml-text
Sep 8, 2023
Merged

Add standard {ReadFrom,WriteTo}{Csv,Json} yaml endpoints.#28268
robertwb merged 10 commits into
apache:masterfrom
robertwb:yaml-text

Conversation

@robertwb

Copy link
Copy Markdown
Contributor

Uses infrastructure from #28169 which should be reviewed first.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

* Use the schema provided, rather than just re-ordering.
* Allow for unspecified optional fields.
* Allow skipping of external transforms with unparsable (e.g. language-specific) schemas.
This allows one to remap configs for a cosistant API across languages
(including differences like camelCase vs. snake_case).
This can be invoked from either Python or Java.
@robertwb

Copy link
Copy Markdown
ContributorAuthor

R: @Polber

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@codecov

codecovBot commented Sep 1, 2023

Copy link
Copy Markdown

Codecov Report

Merging #28268 (4007732) into master (45ea758) will increase coverage by 0.05%.
Report is 75 commits behind head on master.
The diff coverage is 56.68%.

❗ Current head 4007732 differs from pull request most recent head b762fb9. Consider uploading reports for the commit b762fb9 to get more accurate results

@@ Coverage Diff @@## master #28268 +/- ##
==========================================
+ Coverage 72.30% 72.35% +0.05% 
==========================================
Files 678 680 +2 Lines 99853 100340 +487 ==========================================
+ Hits 72197 72605 +408 - Misses 26077 26156 +79 
Partials 1579 1579 
FlagCoverage Δ
python82.88% <56.68%> (+<0.01%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files ChangedCoverage Δ
sdks/python/apache_beam/yaml/main.py0.00% <0.00%> (ø)
sdks/python/apache_beam/io/gcp/bigquery.py69.54% <17.24%> (-0.84%)⬇️
sdks/python/apache_beam/yaml/yaml_io.py48.57% <48.57%> (ø)
sdks/python/apache_beam/pipeline.py91.86% <50.00%> (-0.27%)⬇️
sdks/python/apache_beam/yaml/yaml_transform.py89.75% <50.00%> (-0.18%)⬇️
sdks/python/apache_beam/transforms/external.py74.36% <70.73%> (-0.23%)⬇️
sdks/python/apache_beam/yaml/yaml_provider.py71.88% <82.85%> (+1.92%)⬆️
...ks/python/apache_beam/typehints/schema_registry.py96.15% <100.00%> (+0.91%)⬆️

... and 18 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@robertwb

Copy link
Copy Markdown
ContributorAuthor

Run Java_GCP_IO_Direct PreCommit

@robertwb
robertwb merged commit ee8e53b into apache:masterSep 8, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@robertwb@Polber