Suggest marking ``test_free_threading.test_dict`` as a heavy-cpu test

[Eclips4]

Bug description:

On my machine (macbook air M1) single run of test_free_threading takes about two minutes:

./python.exe -m test -q test_free_threading
Using random seed: 1314296093
0:00:00 load avg: 1.62 Run 1 test sequentially
test_free_threading passed in 1 min 42 sec

== Tests result: SUCCESS ==

Total duration: 1 min 42 sec
Total tests: run=19
Total test files: run=1/1
Result: SUCCESS

Refleak tests takes much longer time:

./python.exe -m test -R 3:3 test_free_threading
Using random seed: 1878526610
0:00:00 load avg: 2.82 Run 1 test sequentially
0:00:00 load avg: 2.82 [1/1] test_free_threading
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XX. ...
test_free_threading passed in 9 min 54 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 9 min 54 sec
Total tests: run=19
Total test files: run=1/1
Result: SUCCESS

But if we apply requires("cpu") decorator to the test_dict.TestDict, time for the refleak tests reduces to ~3 minutes:

./python.exe -m test -R 3:3 test_free_threading
Using random seed: 2985174183
0:00:00 load avg: 1.94 Run 1 test sequentially
0:00:00 load avg: 1.94 [1/1] test_free_threading
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XXX .2.
test_free_threading leaked [0, 2, 0] memory blocks, sum=2 (this is fine)
test_free_threading passed in 2 min 54 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 54 sec
Total tests: run=14 skipped=1
Total test files: run=1/1
Result: SUCCESS

FYI, this is a debug, non free-threaded build.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-119599: Mark test_free_threading.test_dict as cpu-heavy test #120565

[Eclips4]

Windows, debug non free-threaded build:

./python -m test -q test_free_threading
Running Debug|x64 interpreter...
Using random seed: 2075947594
0:00:00 Run 1 test sequentially
test_free_threading passed in 6 min 15 sec

== Tests result: SUCCESS ==

Total duration: 6 min 15 sec
Total tests: run=19
Total test files: run=1/1
Result: SUCCESS

With a require("cpu") decorator:

./python -m test -q test_free_threading
Running Debug|x64 interpreter...
Using random seed: 3746090352
0:00:00 Run 1 test sequentially
test_free_threading passed in 1 min 40 sec

== Tests result: SUCCESS ==

Total duration: 1 min 40 sec
Total tests: run=14 skipped=1
Total test files: run=1/1
Result: SUCCESS

Linux (WSL) debug, non free-threaded build:

./python -m test -q test_free_threading
Using random seed: 744109109
0:00:00 load avg: 16.23 Run 1 test sequentially
test_free_threading passed in 1 min 17 sec

== Tests result: SUCCESS ==

Total duration: 1 min 17 sec
Total tests: run=19
Total test files: run=1/1
Result: SUCCESS

With applied requires("cpu") decorator:

Using random seed: 2007576896
0:00:00 load avg: 1.94 Run 1 test sequentially

== Tests result: SUCCESS ==

Total duration: 18.3 sec
Total tests: run=14 skipped=1
Total test files: run=1/1
Result: SUCCESS

[colesbury]

@DinoV, can you adjust these tests?

I'd rather we reduce the test iterations to make them faster instead of marking them as "cpu". We get much better signal from tests in GitHub actions CI than in buildbots.

We can also further reduce the iterations (or consider skipping the tests entirely) in the default build.

[nineteendo]

Well, they do require the cpu currently and that slows down all workflows on GitHub. I think it's best to mark them as such until they run fast enough.

[nineteendo]

@vstinner, could you close this?

[colesbury]

@nineteendo made test_free_threading only run on the free-threaded build instead in:

• Skip test_free_threading with GIL #120659

Thanks @nineteendo for following up on this.