[3.12] gh-110276: No longer ignore PROFILE_TASK failure silently (#110295)

[vstinner]

(cherry picked from commit 6ab6040)

• Issue: test_unicode always fails during a PGO-optimised build (because it does not exist) #110276

---

📚 Documentation preview 📚: https://cpython-previews--111950.org.readthedocs.build/

[vstinner]

We should not ignore test failure when running the PGO build: see the new failure #111929 impacting 3.11 and 3.12 branches.

[vstinner]

Once merged, this change should be backported to the 3.11 branch once #111949 is merged.

[vstinner]

cc @AlexWaygood

[vstinner]

@Yhg1s: Are you ok with this change? Previously, the build couldn't fail. But in exchange, we can miss real bugs. See #110276 and #111929 for two recent PGO bugs.

[vstinner]

Oh, the initial issue was #101634

[vstinner]

@gpshead: 3.11 and 3.12 are also affected. Are you fine with my change?

[gpshead]

yeah backporting that makes sense.

[Yhg1s]

No, I would rather not backport this change. The job of the profiling run isn't to check whether tests pass or not. It's to exercise enough of Python and the standard library to make a reasonable profile. A test failure doesn't mean that didn't happen. Even not running any tests doesn't mean there is no reasonable profile. I don't think it's enough of a problem to warrant breaking existing 3.12 build processes.

Making the profile task test failures fatal in 3.13 is fine, but I'd rather not disrupt anyone's existing 3.12 builds and make upgrading to 3.12.1 that much harder.

[vstinner]

@Yhg1s:

Making the profile task test failures fatal in 3.13 is fine, but I'd rather not disrupt anyone's existing 3.12 builds and make upgrading to 3.12.1 that much harder.

@AlexWaygood: Python 3.12 has spoken, I close this PR ;-)

[vstinner]

If tomorrow, this issue becomes a major issue, maybe we can consider adding an option to regrtest to return a non-zero exit code if --pgo tests fail. I did that for "env changed": I added --fail-env-changed, option enabled by --fast-ci and --slow-ci used by buildbots and pre-commit CIs. By default, "env changed" are not reported as errors (by default, exit code is zero: success).