Skip to content

Simple provision to help dealing with flakey CI jobs (repeat_command.py) - #3106

Closed
rwgk wants to merge 12 commits into
pybind:masterfrom
rwgk:test_iostream_num_runs
Closed

Simple provision to help dealing with flakey CI jobs (repeat_command.py)#3106
rwgk wants to merge 12 commits into
pybind:masterfrom
rwgk:test_iostream_num_runs

Conversation

@rwgk

@rwgkrwgk commented Jul 13, 2021

Copy link
Copy Markdown
Collaborator

Experiments trying to understand the MSVC test_iostream.py flakiness.

@rwgk
rwgkforce-pushed the test_iostream_num_runs branch 3 times, most recently from fa56549 to a0584bcCompareJuly 15, 2021 18:51
@rwgk
rwgkforce-pushed the test_iostream_num_runs branch from a0584bc to 4369781CompareJuly 16, 2021 00:37
@rwgkrwgk changed the title Experiments: test_iostream.py NUM_RUNSSimple provision to help dealing with flakey CI jobs (repeat_command.py)Jul 16, 2021
@Skylion007

Copy link
Copy Markdown
Collaborator

There is also pytest-flake which could be useful here.

@rwgk

rwgk commented Aug 7, 2021

Copy link
Copy Markdown
CollaboratorAuthor

If anyone wants to use my trick to deflake tests in the future, PR #3180 should make it straightforward to pin-point where the tests are run, and repeat_command.py now lives here: https://github.com/rwgk/rwgk_config/tree/master/bin

Permalink: https://github.com/rwgk/rwgk_config/blob/e8396e6126741b8916dae91b3024fa0d713e2cea/bin/repeat_command.py

The only slight wrinkle is that one ci.yml job runs the Python tests outside the pybind11 source directory. This is what I did for that one special case:

 - name: Copy tools/repeat_command.py to /tmp
run: cp -a tools/repeat_command.py /tmp
...
run: python3 /tmp/repeat_command.py 1 make pytest -j 2

@rwgkrwgk closed this Aug 7, 2021
@rwgk
rwgk deleted the test_iostream_num_runs branch August 7, 2021 14:53
@rwgkrwgk mentioned this pull request Feb 10, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@rwgk@Skylion007