Skip to content

[BEAM-12164]: fix the autoscaling backlog estimation for Spanner Change Streams Connector - #17200

Merged
pabloem merged 5 commits into
apache:masterfrom
hengfengli:update-restriction-tracker
Apr 14, 2022
Merged

[BEAM-12164]: fix the autoscaling backlog estimation for Spanner Change Streams Connector#17200
pabloem merged 5 commits into
apache:masterfrom
hengfengli:update-restriction-tracker

Conversation

@hengfengli

@hengfenglihengfengli commented Mar 29, 2022

Copy link
Copy Markdown
Contributor

No description provided.

thiagotnunes
thiagotnunes reviewed Mar 29, 2022
View reviewed changes
Comment thread...in/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/QueryChangeStreamAction.java Outdated
Comment thread...ava/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java Outdated
Comment thread...ava/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java Outdated
Comment thread...n/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimator.java Outdated
Comment thread...va/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimatorTest.java Outdated
Comment thread.../src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/util/TimestampGenerator.java Outdated
Comment thread...n/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimator.java Outdated
@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch 4 times, most recently from d9031fe to 06fd55cCompareMarch 31, 2022 06:05
@codecov

codecovBot commented Mar 31, 2022•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #17200 (8af69d7) into master (0086a2b) will increase coverage by 0.02%.
The diff coverage is n/a.

❗ Current head 8af69d7 differs from pull request most recent head 8c990db. Consider uploading reports for the commit 8c990db to get more accurate results

@@ Coverage Diff @@## master #17200 +/- ##
==========================================
+ Coverage 73.94% 73.97% +0.02% 
==========================================
Files 684 672 -12 Lines 89519 88460 -1059 ==========================================
- Hits 66194 65437 -757 + Misses 22165 21902 -263 + Partials 1160 1121 -39
FlagCoverage Δ
go49.56% <0.00%> (-0.21%)⬇️
python83.66% <0.00%> (+0.02%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted FilesCoverage Δ
sdks/go/pkg/beam/core/runtime/graphx/cogbk.go0.00% <0.00%> (-72.73%)⬇️
...beam/runners/portability/local_job_service_main.py0.00% <0.00%> (-14.44%)⬇️
sdks/go/pkg/beam/core/runtime/graphx/translate.go28.81% <0.00%> (-14.12%)⬇️
sdks/python/apache_beam/runners/job/utils.py69.23% <0.00%> (-7.70%)⬇️
sdks/go/pkg/beam/runners/dataflow/dataflow.go50.00% <0.00%> (-3.03%)⬇️
sdks/python/apache_beam/io/gcp/gcsio.py90.12% <0.00%> (-2.50%)⬇️
...ache_beam/runners/portability/local_job_service.py80.00% <0.00%> (-1.58%)⬇️
sdks/go/pkg/beam/core/graph/coder/map.go63.72% <0.00%> (-1.28%)⬇️
sdks/go/pkg/beam/core/runtime/exec/translate.go5.81% <0.00%> (-1.16%)⬇️
sdks/python/apache_beam/internal/metrics/metric.py91.00% <0.00%> (-1.00%)⬇️
... and 125 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0086a2b...8c990db. Read the comment docs.

@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch from 06fd55c to 3c952c4CompareApril 4, 2022 02:22
@hengfengli

hengfengli commented Apr 4, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java_Examples_Dataflow PreCommit

@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch 3 times, most recently from 68fa94e to fd1e90aCompareApril 4, 2022 07:18
thiagotnunes
thiagotnunes reviewed Apr 6, 2022
View reviewed changes
Comment thread...ava/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java Outdated
Comment thread...n/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimator.java Outdated
Comment thread...c/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/TimestampUtils.java Outdated
Comment thread...va/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimatorTest.java Outdated
@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch from 194eea9 to a55cb4bCompareApril 6, 2022 09:19
thiagotnunes
thiagotnunes approved these changes Apr 7, 2022
View reviewed changes

@thiagotnunesthiagotnunes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread...va/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimatorTest.java Outdated
@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch 2 times, most recently from 17a2d10 to 2afbc65CompareApril 10, 2022 23:49
@hengfengli

hengfengli commented Apr 11, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@thiagotnunesthiagotnunes mentioned this pull request Apr 12, 2022
Merged
@pabloem

pabloem commented Apr 12, 2022

Copy link
Copy Markdown
Member

taking a look at this...

@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch from dc2cbcc to 96362e2CompareApril 12, 2022 23:24
@thiagotnunes

thiagotnunes commented Apr 12, 2022

Copy link
Copy Markdown
Contributor

Run Java PostCommit

@hengfengli
hengfengliforce-pushed the update-restriction-tracker branch from 5b03701 to 8c990dbCompareApril 13, 2022 05:07
@hengfengli

hengfengli commented Apr 13, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@thiagotnunes

thiagotnunes commented Apr 13, 2022

Copy link
Copy Markdown
Contributor

Run Java PostCommit

@pabloem

pabloem commented Apr 14, 2022

Copy link
Copy Markdown
Member

this LGTM. Thanks y'all!

@pabloem
pabloem merged commit 8a33871 into apache:masterApr 14, 2022
@TheNeuralBitTheNeuralBit mentioned this pull request Oct 6, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pabloempabloempabloem approved these changes
+1 more reviewer
@thiagotnunesthiagotnunesthiagotnunes approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

dataflowgcpiojavarunners

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hengfengli@pabloem@thiagotnunes