`test___all__.AllTest.test_all` is failing on every PR

[AlexWaygood]

Bug report

Following #105801, test___all__.AllTest.test_all is failing on every non-docs PR on the "Hypothesis tests on Ubuntu" CI job.

The traceback is:

======================================================================
FAIL: test_all (test.test___all__.AllTest.test_all)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test___all__.py", line 129, in test_all
 self.check_all(modname)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test___all__.py", line 43, in check_all
 with warnings_helper.check_warnings(
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/contextlib.py", line 144, in __exit__
 next(self.gen)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/warnings_helper.py", line 185, in _filterwarnings
 raise AssertionError("unhandled warning %s" % reraise[0])
AssertionError: unhandled warning {message : SyntaxWarning("invalid escape sequence '\\?'"), category : 'SyntaxWarning', filename : '/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_httpservers.py', lineno : 445, line : None}

Example failures:

• https://github.com/python/cpython/actions/runs/5277208289/jobs/9544938249
• https://github.com/python/cpython/actions/runs/5274715744/jobs/9539462432?pr=105794
• https://github.com/python/cpython/actions/runs/5272447172/jobs/9534727485

This failure was visible in the CI for #105801 FYI

Linked PRs

• gh-105821: Use a raw f-string in test_httpservers.py #105822
• [3.12] gh-105821: Use a raw f-string in test_httpservers.py (GH-105822) #105824
• [3.11] gh-105821: Use a raw f-string in test_httpservers.py (GH-105822) #108576
• [3.10] gh-105821: Use a raw f-string in test_httpservers.py (GH-105822) #115519

[pablogsal]

This failure was visible in the CI for #105801 FYI

But then how is possible that we could land that PR?

[pablogsal]

Ah, because hypothesis tests are not mandatory. Well, this is what I feared that would happen having non-mandatory tests that are already failing for other reasons 😞

[AlexWaygood]

Ah, because hypothesis tests are not mandatory.

Indeed

Well, this is what I feared that would happen having non-mandatory tests that are already failing for other reasons 😞

FWIW, I haven't seen any failures on the "hypothesis tests" job since @ericsnowcurrently fixed the test__xxsubinterpreter and test_threading crashes. We could probably make it a "required" CI check at this point.

[pablogsal]

We could probably make it a "required" CI check at this point.

I think we should

[AlexWaygood]

We could probably make it a "required" CI check at this point.

I think we should

We should probably continue discussion on the matter at #86275