Test suite crashes when run single process `--with-pydebug` enabled

[freakboy3742]

Crash report

What happened?

The make testios testing target crashes when compiled --with-pydebug enabled (as is done in CI).

The error is a SIGABRT, raised during test_types; however, running test_types by itself isn't enough to reproduce the problem.

The error is raised on L8476 of typeobject.c:

 assert(self->tp_version_tag != 0);

The same failure can be manufactured on macOS with M1 hardware, as long as the test suite is executed as a single process (i.e. python -m test, not the multi-process option enabled by make test).

Full error trace on iOS:
stacktrace.txt

The same problem doesn't appear to occur when debug is not enabled.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS, iOS

Output from running 'python -VV' on the command line:

3.14.0a0; verified on 2bac2b86, but possibly present back to dc03ce7

Linked PRs

• gh-121832: Skip subinterpreter static type check on iOS to restore test suite. #122150
• [3.13] gh-121832: Skip subinterpreter static type check on iOS to restore test suite. (GH-122150) #122159
• GH-121832: Assert that the version number of static builtin types is not changed by PyType_Modified. #122182
• [3.13] GH-121832: Assert that the version number of static builtin types is not changed by PyType_Modified. (GH-122182) #122290
• gh-121832: Revert test skip introduced by #122150. #122340
• [3.13] gh-121832: Revert test skip introduced by GH-122150. (GH-122340) #122342

[freakboy3742]

I've narrowed the cause of the crash to #121602 (which was backported to 3.13 as #121632, so it explains the failures on both branches).

[freakboy3742]

I have confirmed the crash is occurring in the SubInterpreterTest added as part of #121602. If that test is commented out, the test suite passes without error.

[ericsnowcurrently]

FYI, I'm pretty sure gh-121636 will fix this.

The gist is: the change in gh-121602 runtime initialization makes a copy of the (mostly) unmodified PyTypeObject struct for each builtin static type. During finalization we restore that copy, including the tp_version_tag. Clearly a part of that is causing things to get in an inconsistent state on iOS. The fix in gh-121636 restricts the copy/restore to only the "tp" slots (function pointers), restoring the pre-121602 status quo for the rest, including tp_version_tag.

Is there a way you could verify gh-121636 fixes the iOS build?

All that said, I'm definitely curious why this would be manifesting only on iOS?

[freakboy3742]

FYI, I'm pretty sure gh-121636 will fix this.
...
Is there a way you could verify gh-121636 fixes the iOS build?

The iOS buildbot should be sufficient to verify any fix; from the look of it, the current state of #121636 doesn't fix it. The test failure isn't very helpful - it's a hard crash of the emulator, which the test report doesn't show very nicely - but with the test_types SubinterpreterTest commented out, the suite completes and passes on iOS. I've verified the same tests locally, and I get the same failure on #121636.

All that said, I'm definitely curious why this would be manifesting only on iOS?

You and me both :-)

My best guess right now it's not iOS specific, but a function of how the test suite is executed on iOS. iOS is forced to run the entire test suite sequentially in a single process. This isn't (AFAIK) the default mode of operation on any other runner; so it's possible the specific sequence of test execution is causing an odd state to emerge. One of the debugging tasks I'm going to look into today is to try and reproduce this on macOS (the other is to try and find a subset of tests that causes the failure, rather than needing to get to test_types alphabetically... after 20 minutes)

[ericsnowcurrently]

Looks like gh-121636 isn't sufficient. I'm going to try a different approach.

[ericsnowcurrently]

See gh-121882.

[freakboy3742]

Dang it... looks like gh-121882 doesn't fix the problem either... 😢

[freakboy3742]

Confirming: the test_types test doesn't fail on macOS when the test suite is executed in single process mode, so the problem isn't purely about execution order. Now trying to bisect the test suite to find a smaller subset of tests that fail.