Skip to content

[BEAM-11719 ] Use deterministic coders for grouping keys. - #13839

Merged
robertwb merged 8 commits into
apache:masterfrom
robertwb:structural-keys
Feb 25, 2021
Merged

[BEAM-11719 ] Use deterministic coders for grouping keys.#13839
robertwb merged 8 commits into
apache:masterfrom
robertwb:structural-keys

Conversation

@robertwb

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

LangSDKDataflowFlinkSamzaSparkTwister2
GoBuild Status---Build Status---Build Status---
JavaBuild StatusBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild Status
Build Status
Build Status
Build Status
PythonBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
---Build Status---
XLangBuild StatusBuild StatusBuild Status---Build Status---

Pre-Commit Tests Status (on master branch)

---JavaPythonGoWebsiteWhitespaceTypescript
Non-portableBuild StatusBuild Status
Build Status
Build Status
Build Status
Build StatusBuild StatusBuild StatusBuild Status
Portable---Build Status------------

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@robertwbrobertwb changed the title Use deterministic coders for grouping keys.[BEAM-11719 ] Use deterministic coders for grouping keys.Feb 1, 2021
@codecov

codecovBot commented Feb 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #13839 (1318fbd) into master (c137e88) will increase coverage by 0.01%.
The diff coverage is n/a.

Impacted file tree graph

@@ Coverage Diff @@## master #13839 +/- ##
==========================================
+ Coverage 82.97% 82.99% +0.01% 
==========================================
Files 469 469 Lines 58300 58325 +25 ==========================================
+ Hits 48377 48407 +30 + Misses 9923 9918 -5 
Impacted FilesCoverage Δ
.../py38/build/srcs/sdks/python/apache_beam/pvalue.py
...ks/python/apache_beam/internal/metrics/__init__.py
...eam/runners/portability/fn_api_runner/execution.py
...hon/apache_beam/examples/cookbook/mergecontacts.py
...s/python/apache_beam/io/gcp/bigquery_file_loads.py
...testing/benchmarks/nexmark/queries/winning_bids.py
...dks/python/apache_beam/metrics/monitoring_infos.py
...ild/srcs/sdks/python/apache_beam/io/gcp/dicomio.py
.../apache_beam/runners/dataflow/internal/__init__.py
...python/apache_beam/runners/portability/__init__.py
... and 928 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f647b05...1318fbd. Read the comment docs.

@robertwb

Copy link
Copy Markdown
ContributorAuthor

R: @udim
CC: @katsiapis

Comment threadsdks/python/apache_beam/transforms/ptransform_test.py Outdated
Comment threadsdks/python/apache_beam/transforms/ptransform_test.py Outdated
Comment threadsdks/python/apache_beam/transforms/ptransform_test.py Outdated
Comment threadsdks/python/apache_beam/runners/dataflow/dataflow_runner.py
deleting_tables: deleting_tables,
pvalue.AsIter(temp_tables_pc))
| "RemoveTempTables/AddUselessValue" >> beam.Map(lambda x: (x, None))
# TableReference has no deterministic coder, but as this de-duplication

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can tell, parse_table_reference is used to normalize and populate TableReference instances. They are essentially a "(project, dataset, table) tuple."
It should be possible to beam.Map(lambda x: (x.tableId, x.datasetId, x.projectId)) | beam.Distinct() | beam.Map(lambda x: parse_table_reference(*x))

cc: @pabloem

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming exactly these three fields would not be as future proof.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. The official BigQuery client (which this code doesn't use) has a hashable wrapper for this.
https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.table.TableReference.html

Comment threadsdks/python/apache_beam/runners/pipeline_context.py

@robertwbrobertwb left a comment

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, PTAL.

deleting_tables: deleting_tables,
pvalue.AsIter(temp_tables_pc))
| "RemoveTempTables/AddUselessValue" >> beam.Map(lambda x: (x, None))
# TableReference has no deterministic coder, but as this de-duplication

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming exactly these three fields would not be as future proof.

Comment threadsdks/python/apache_beam/runners/pipeline_context.py
Comment threadsdks/python/apache_beam/transforms/ptransform_test.py Outdated
Comment threadsdks/python/apache_beam/transforms/ptransform_test.py Outdated
@robertwb

Copy link
Copy Markdown
ContributorAuthor

Run Python_PVR_Flink PreCommit

@chunyang

Copy link
Copy Markdown
Contributor

I seem to be bumping into an error caused by deterministic coder enforcement. Do you mind taking a look at ##14112?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@robertwb@chunyang@udim