gh-135410: Use a critical section around StringIO.__next__

[ZeroIntensity]

I'm generally not a fan of adding locks around __next__, because concurrent iteration isn't generally a real use case, but readlines calls into __next__.

• Issue: Segfault from calling StringIO methods in threads on free-threading debug build #135410

[corona10]

Let's consider the following options.

1. Prevent segfault only.
2. Guarantee thread safety also. But IIRC, we do not guarantee all stdlib iterations.

[ZeroIntensity]

Yeah, I totally agree on that. There was an issue similar to this a few weeks ago, where a TextIOWrapper.__next__ didn't have a critical section on it, which ended up causing crashes. I originally rejected it, but I was eventually convinced that it is worth doing for IO objects.

[corona10]

I am fine with if this change will not regress overall performance. But if the object is shared by multiple threads user should care about the correctness if the object itself does not gurentee it.
cc @colesbury

[vstinner]

LGTM.

Without the fix, the test fails randomly with:

FAIL: test_concurrent_use (test.test_memoryio.PyStringIOPickleTest.test_concurrent_use)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vstinner/python/main/Lib/test/test_memoryio.py", line 742, in test_concurrent_use
self.assertIsNone(cm.exc_value)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

So the test works as expected!

[vstinner]

But if the object is shared by multiple threads user should care about the correctness if the object itself does not gurentee it.

Currently, the doc says nothing about io.StringIO: https://docs.python.org/dev/library/io.html#multi-threading

[miss-islington-app]

Thanks @ZeroIntensity for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @ZeroIntensity for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @ZeroIntensity and @kumaraditya303, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker e6c3039cb39e68ae9af9ddcaca341c5af8f9cf23 3.13

[bedevere-app]

GH-135425 is a backport of this pull request to the 3.14 branch.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL 3.14 (tier-1) has failed when building commit 8cb7d9a.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1733/builds/261) and take a look at the build logs.
4. Check if the failure is related to this commit (8cb7d9a) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1733/builds/261

Failed tests:

• test_memoryio

Failed subtests:

• test_concurrent_use - test.test_memoryio.PyStringIOPickleTest.test_concurrent_use

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
File "/home/buildbot/buildarea/3.14.itamaro-centos-aws.nogil/build/Lib/test/test_memoryio.py", line 742, in test_concurrent_useself.assertIsNone(cm.exc_value)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit e6c3039.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/1566) and take a look at the build logs.
4. Check if the failure is related to this commit (e6c3039) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/1566

Failed tests:

• test_memoryio

Failed subtests:

• test_concurrent_use - test.test_memoryio.PyStringIOPickleTest.test_concurrent_use
• test_concurrent_use - test.test_memoryio.PyStringIOTest.test_concurrent_use

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_memoryio.py", line 742, in test_concurrent_useself.assertIsNone(cm.exc_value)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.14 (tier-1) has failed when building commit 8cb7d9a.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1774/builds/185) and take a look at the build logs.
4. Check if the failure is related to this commit (8cb7d9a) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1774/builds/185

Failed tests:

• test_memoryio

Failed subtests:

• test_concurrent_use - test.test_memoryio.PyStringIOPickleTest.test_concurrent_use
• test_concurrent_use - test.test_memoryio.PyStringIOTest.test_concurrent_use

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
File "/home/buildbot/buildarea/3.14.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_memoryio.py", line 742, in test_concurrent_useself.assertIsNone(cm.exc_value)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

[ZeroIntensity]

Those buildbot failures look related. I'll quickly investigate and if I can't figure it out in the next half an hour or so, I'll put up a revert.

[vstinner]

IMO this change was merged too quickly and I suggest to revert it.

• I was awaiting for @colesbury to review the change.
• @corona10 asked for benchmarks, this request was ignored.
• I pointed out that the documentation says nothing about io.StringIO thread safety. I would prefer to address this point at the same time.
• And now, Refleak buildbots are failing. It seems like the Python implementation (_pyio) behaves differently than the C implementation (io).

I suggest to revert the two changes (main and 3.14 branches) and spend more time to design this fix. I don't think that there is any urgency to fix this issue.

[ZeroIntensity]

Sorry for the hassle everyone! I didn't get a chance to reply to any comments before it was merged. I'll put up a revert while we discuss this.