Skip to content

Make num-stages counter into an internal counter. - #16638

Merged
ibzib merged 1 commit into
apache:masterfrom
robertwb:internal-counters
Feb 2, 2022
Merged

Make num-stages counter into an internal counter.#16638
ibzib merged 1 commit into
apache:masterfrom
robertwb:internal-counters

Conversation

@robertwb

Copy link
Copy Markdown
Contributor

This unbreaks tests that (arguably to brittly) reject extra counters.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

LangULRDataflowFlinkSamzaSparkTwister2
Go---Build StatusBuild StatusBuild StatusBuild Status---
JavaBuild StatusBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python---Build Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild Status---
XLangBuild StatusBuild Status
Build Status
Build Status
Build StatusBuild StatusBuild Status---

Examples testing status on various runners

LangULRDataflowFlinkSamzaSparkTwister2
Go---------------------
Java---Build Status
Build Status
Build Status
---------------
Python---------------------
XLang---------------------

Post-Commit SDK/Transform Integration Tests Status (on master branch)

GoJavaPython
Build StatusBuild StatusBuild Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

---JavaPythonGoWebsiteWhitespaceTypescript
Non-portableBuild Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild StatusBuild StatusBuild Status
Portable---Build StatusBuild Status---------

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

This unbreaks tests that (arguably to brittly) reject extra counters.
@robertwb

Copy link
Copy Markdown
ContributorAuthor

Follow-up from #16101

CC: @kileys

@codecov

codecovBot commented Jan 27, 2022

Copy link
Copy Markdown

Codecov Report

Merging #16638 (7d3af58) into master (1351030) will increase coverage by 37.19%.
The diff coverage is n/a.

Impacted file tree graph

@@ Coverage Diff @@## master #16638 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19% 
===========================================
Files 201 452 +251 Lines 19787 62095 +42308 ===========================================
+ Hits 9189 51931 +42742 - Misses 9612 10164 +552 + Partials 986 0 -986 
Impacted FilesCoverage Δ
...eam/runners/portability/fn_api_runner/fn_runner.py90.90% <ø> (ø)
sdks/go/pkg/beam/pardo.go
sdks/go/pkg/beam/partition.go
...pkg/beam/core/runtime/xlangx/expansionx/process.go
sdks/go/pkg/beam/core/runtime/genx/genx.go
sdks/go/pkg/beam/core/metrics/store.go
...ks/go/pkg/beam/core/runtime/harness/cache_hooks.go
sdks/go/pkg/beam/core/runtime/metricsx/urns.go
sdks/go/pkg/beam/core/runtime/graphx/serialize.go
sdks/go/pkg/beam/core/util/hooks/hooks.go
... and 644 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1351030...7d3af58. Read the comment docs.

@robertwb

Copy link
Copy Markdown
ContributorAuthor

Run Python PreCommit

str(type(self)),
self.NUM_FUSED_STAGES_COUNTER,
urn='internal:' + self.NUM_FUSED_STAGES_COUNTER)).update(
len(stages))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess internal: is a concept already defined somewhere?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, there's some inconsistencies here with "user" counters requiring a name and namespace and "non-user" counters using the first component of the urn as a namespace and the rest as the name.

@robertwb

Copy link
Copy Markdown
ContributorAuthor

The failing tests

apache_beam.transforms.create_test.CreateTest.test_create_uses_coder_for_pickling
apache_beam.transforms.combinefn_lifecycle_test.LocalCombineFnLifecycleTest_0.test_combine

look unrelated. Trying again.

@robertwb

Copy link
Copy Markdown
ContributorAuthor

Run Python PreCommit

@robertwb

Copy link
Copy Markdown
ContributorAuthor

@TheNeuralBit

Copy link
Copy Markdown
Member

Run Python PreCommit

@ibzib

ibzib commented Feb 2, 2022

Copy link
Copy Markdown

I tried deflaking the Actions tests but it's unlikely to work since the rate of flakes is so high, there's no way to re-run individual actions, and the flakes can affect 8 different actions... I'm just going to merge this

@ibzib
ibzib merged commit c7a7eec into apache:masterFeb 2, 2022
@TheNeuralBit

Copy link
Copy Markdown
Member

Thanks @ibzib, I think that the flakiness is actually resolved at HEAD, but for some reason github's /merge ref is pointing to an older commit cf79c26

image

@TheNeuralBit

Copy link
Copy Markdown
Member

I suppose the merge ref is likely just created once when the PR is created and not updated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@robertwb@TheNeuralBit@ibzib