Skip to content

gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') - #108480

Merged
serhiy-storchaka merged 4 commits into
python:mainfrom
serhiy-storchaka:mark-slow-tests-walltime
Sep 5, 2023
Merged

gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime')#108480
serhiy-storchaka merged 4 commits into
python:mainfrom
serhiy-storchaka:mark-slow-tests-walltime

Conversation

@serhiy-storchaka

@serhiy-storchakaserhiy-storchaka commented Aug 25, 2023

Copy link
Copy Markdown
Member

@serhiy-storchaka

Copy link
Copy Markdown
MemberAuthor

These are mostly tests which contain several sleep()s or call a function which expected to fail by timeout. There are few tests which depend on slow external network service.

@serhiy-storchaka

Copy link
Copy Markdown
MemberAuthor
$ ./python -m test -uall,-cpu -j4
...
Total duration: 7 min 27 sec
Total tests: run=42,127 skipped=1,077
$ ./python -m test -uall,-cpu,-walltime -j4
...
Total duration: 6 min 26 sec
Total tests: run=42,127 skipped=1,139

@miss-islington

Copy link
Copy Markdown
Contributor

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

@serhiy-storchaka
serhiy-storchaka deleted the mark-slow-tests-walltime branch September 5, 2023 14:56
@miss-islington

Copy link
Copy Markdown
Contributor

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 1e0d62793a84001e92f1c80b511d3a212b435acc 3.11

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 5, 2023
…es_resource('walltime') (pythonGH-108480)
(cherry picked from commit 1e0d627)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot

Copy link
Copy Markdown

GH-108923 is a backport of this pull request to the 3.12 branch.

@bedevere-botbedevere-bot removed the needs backport to 3.12 only security fixes label Sep 5, 2023
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Sep 5, 2023
… requires_resource('walltime') (pythonGH-108480).
(cherry picked from commit 1e0d627)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot

Copy link
Copy Markdown

GH-108924 is a backport of this pull request to the 3.11 branch.

@bedevere-botbedevere-bot removed the needs backport to 3.11 only security fixes label Sep 5, 2023
serhiy-storchaka added a commit that referenced this pull request Sep 5, 2023
vstinner pushed a commit to vstinner/cpython that referenced this pull request Sep 5, 2023
Yhg1s pushed a commit that referenced this pull request Sep 8, 2023
* gh-108834: regrtest reruns failed tests in subprocesses (#108839)
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.
Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.
Changes:
* Fix and enhance statistics in regrtest summary. Add "(filtered)"
when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
(cherry picked from commit 31c2945)
* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
(cherry picked from commit 1170d5a)
* gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480)
(cherry picked from commit 1e0d627)
* Manually sync Lib/test/libregrtest/ from main
---------
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@serhiy-storchakaserhiy-storchaka removed their assignment Sep 8, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip newstestsTests in the Lib/test dir

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@serhiy-storchaka@miss-islington@bedevere-bot@gpshead