bpo-37707: Exclude expensive unit tests from PGO task

[nascheme]

Mark some individual tests to skip when --pgo is used. The tests
marked increase the PGO task time significantly and likely don't
help improve optimization of the final executable.

Marked tests:

pickletester
check_frame_opcodes
test_framing_many_objects
test_bz2
testCompress4G
testDecompress4G
test_itertools
test_long_chain_of_empty_iterables
test_lzma
test_compressor_bigmem
test_decompressor_bigmem
test_statistics
test_inv_cdf

https://bugs.python.org/issue37707

[gpshead]

Nice! Such a pgo skip decorator approach is exactly what I was thinking of recommending when you brought this up. :)

[zooba]

I haven't looked through exactly what these tests do, but I'd say it's worth evaluating ("guessing") whether they represent a common scenario or not. Even if they run for a long time, if it's something that's going to run a lot in user code then I'd rather keep it in.

99% of realistic Python runtime ("evaluated"/"guessed") is success cases, but a lot of the quick tests in the suite are going to exercise setup/teardown and failure paths. Let's not over-focus on those for the sake of a quick PGO run.

[nascheme]

I intended that --pgo will give you an optimized build without too much extra build time. Before PR #14702, the PGO builds took so long that in practice it seems that many did not bother to use them. If you don't care at all about long build time, then --pgo-extended is the option for that. If we make expensive tests run in the --pgo case, I don't see the point of --pgo-extended.

Something I was thinking about previously, --pgo-extended should not not skip the tests marked with @skip_if_pgo_task. That way, people who want the most extensive code coverage can still use --pgo-extended to get the old behavior.

[nascheme]

Here is a comparison of test run time before and after my skip decorator.

test	before [s]	after [s]
test_pickle	4.86	1.79
test_bz2	1.98	0.51
test_itertools	1.51	1.12
test_lzma	2.65	0.19
test_statistics	1.84	0.57

[pablogsal]

I haven't looked through exactly what these tests do, but I'd say it's worth evaluating ("guessing") whether they represent a common scenario or not.

I did two PGO builds with and without this PR and pyperformance shows no significant difference (all differences were <1%). I am using core isolation and affinity for them so I trust the stability of the result but I can repeat them if we need to double check.

Something I was thinking about previously, --pgo-extended should not not skip the tests marked with @skip_if_pgo_task. That way, people who want the most extensive code coverage can still use --pgo-extended to get the old behavior.

+1

[miss-islington]

Thanks @nascheme for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

[bedevere-bot]

GH-15024 is a backport of this pull request to the 3.8 branch.