Skip to content

[BEAM-11971] Fix directrunner timer consistency - #16650

Merged
reuvenlax merged 1 commit into
apache:masterfrom
reuvenlax:fix_directrunner_looping_timer
Feb 3, 2022
Merged

[BEAM-11971] Fix directrunner timer consistency#16650
reuvenlax merged 1 commit into
apache:masterfrom
reuvenlax:fix_directrunner_looping_timer

Conversation

@reuvenlax

@reuvenlaxreuvenlax commented Jan 29, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor
  • Replaces pushedBackTimers with the technique used in FnApiDoFnRunner.
  • Removes buggy use of Concurrent* data structures (those data structures only provide weakly-consistent iteration, which violates assumptions in the DirectRunner)
  • Fix a few bugs associated with looping timers

@reuvenlax
reuvenlaxforce-pushed the fix_directrunner_looping_timer branch 3 times, most recently from 2f4de0b to a8c89baCompareJanuary 29, 2022 23:00
@reuvenlax

reuvenlax commented Jan 30, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@reuvenlax

reuvenlax commented Jan 30, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java_PVR_Flink_Batch PreCommit

@reuvenlaxreuvenlax changed the title Fix directrunner looping timer[BEAM-11971] Fix directrunner timer consistencyJan 30, 2022
@reuvenlax

reuvenlax commented Jan 31, 2022

Copy link
Copy Markdown
ContributorAuthor

R: @rezarokni

@reuvenlax

reuvenlax commented Jan 31, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@reuvenlax
reuvenlax requested a review from je-ikJanuary 31, 2022 22:42
je-ik
je-ik reviewed Feb 1, 2022
View reviewed changes

@je-ikje-ik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing I'm not sure how is how setting looping timer in earlierTimers should work. Could we maybe test that explicitly with a unit test?

Comment thread.../direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java Outdated
Comment threadrunners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java Outdated

@je-ikje-ikFeb 1, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The construction of the timerKey seems repeated, could we wrap it in a helper method?

@reuvenlaxreuvenlaxFeb 1, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleaned up timerKey usage

Comment threadsdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java Outdated
je-ik
je-ik reviewed Feb 2, 2022
View reviewed changes

@je-ikje-ik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
nit: maybe we could slightly improve the stringKey for better readability.

Comment threadrunners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java Outdated
@reuvenlax

reuvenlax commented Feb 2, 2022

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@lukecwik

lukecwik commented Feb 2, 2022

Copy link
Copy Markdown
Member

CC: @kileys

je-ik
je-ik approved these changes Feb 2, 2022
View reviewed changes

@je-ikje-ik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@reuvenlax
reuvenlaxforce-pushed the fix_directrunner_looping_timer branch from 83d976b to 8ea77f9CompareFebruary 3, 2022 17:17
@reuvenlax
reuvenlax merged commit 6814a06 into apache:masterFeb 3, 2022
@ibzib

ibzib commented Feb 7, 2022

Copy link
Copy Markdown

@reuvenlax - we rolled this back because several tests were failing #16748

@reuvenlax

reuvenlax commented Feb 8, 2022 via email

Copy link
Copy Markdown
ContributorAuthor

@damccormdamccorm mentioned this pull request Jun 4, 2022
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@je-ikje-ikje-ik approved these changes
@kennknowleskennknowlesAwaiting requested review from kennknowles

Assignees

No one assigned

Labels

coredirectjavalocalrunners

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@reuvenlax@lukecwik@ibzib@je-ik