gh-142414: Unify UNBOUND in Lib/concurrent/interpreters/_queues.py and Lib/concurrent/interpreters/_crossinterp.py.

[note35]

The fix is aligned with bug reporter's observation.

Test with

1. ./python -m test test_interpreters test_struct (This one triggers OP's issue.)
2. Repro script by Victor.
3. ./python -m test test_interpreters test_concurrent_futures.test_interpreter_pool (This one triggers the forever loop after the initial fix.)

• Issue: ./python -m test test_interpreters test_struct fails with: KeyError: concurrent.interpreters._queues.UNBOUND #142414

[ZeroIntensity]

This needs a news entry, please add one. You can use blurb-it or the blurb command-line tool.

[ZeroIntensity]

I think it would also be good to add a dedicated test for this, similar to Victor's small repro in the original issue. This bug only occurs under some special circumstances in the current test suite, which isn't ideal.

[note35]

I also created another PR to remove singleton - #142553

[ZeroIntensity]

What happened here?

[note35]

./python -m test test_interpreters test_concurrent_futures.test_interpreter_pool Using random seed: 1867170309
0:00:00 load avg: 0.27 Run 2 tests sequentially in a single process
0:00:00 load avg: 0.27 [1/2] test_interpreters
0:00:24 load avg: 0.48 [1/2] test_interpreters passed
0:00:24 load avg: 0.48 [2/2] test_concurrent_futures.test_interpreter_pool
... (running forever without catching queues.QueueEmpty)

deftest_blocking_with_limited_workers(self):
ready=queues.create()
blocker=queues.create()
defrun(taskid, ready, blocker):
ready.put_nowait(taskid)
blocker.get()
numtasks=10futures= []
withself.executor_type(4) asexecutor:
foriinrange(numtasks):
fut=executor.submit(run, i, ready, blocker)
futures.append(fut)
pending=numtaskswhilepending>0:
done=0for_inrange(pending):
try:
ready.get(timeout=1)
# https://github.com/python/cpython/blob/2a820e2b9cf9e470d9f5342019fca3fe7f4ed7bc/Lib/concurrent/interpreters/_queues.py#L261C20-L261C30# test_interpreters test_concurrent_futures.test_interpreter_pool # Before: queue.get() throws interpreters.QueueEmpty# After: queue.get() throws queues.QueueEmpty# test_concurrent_futures.test_interpreter_pool # Before: queue.get() throws interpreters.QueueEmpty.# After: queue.get() throws interpreters.QueueEmptyexceptinterpreters.QueueEmpty:
passexceptqueues.QueueEmpty:
passelse:
# If the error is not caught, the loop is running forever.done+=1pending-=donefor_inrange(done):
blocker.put_nowait(None)

Why does running test_interpreters + test_concurrent_futures.test_interpreter_pool make the queue.get() throws queues.QueueEmpty after having the same UNBOUND object in _queues.py and _crossinterp.py?

FWIK, cpython/Lib/concurrent/interpreters/init.py loads queues.QueueEmpty as interpreters.QueueEmpty, and before this PR, queue.get() is expected to throw interpreters.QueueEmpty,

For some reasons, queue.get() throws _queues.QueueEmpty after running test_interpreters. This may be another thing to explore further (I think it depends on how the sub-interpreter is implemented. Do you have any knowledge about this?

Given this is a test fix, maybe it's okay to have a further dive deep separately? I think this so far can be reproduced only during a combination of test_interpreters + test_concurrent_futures.test_interpreter_pool. (which is likely rare and not reported?) I'd also like to add a TODO gh-XXX, but we don't yet have enough context to make an issue...)

[encukou]

This looks like a good step forward, even if it found another issue.
@ZeroIntensity, are you OK merging this & leaving QueueEmpty for later?

[ZeroIntensity]

The QueueEmpty change makes me very nervous. I can't tell if that's a new problem introduced by this PR, or just an existing issue unveiled.

If you want to merge this to get tests passing, I think that's fine. Long term, something like #142577 or #145184 will work better.

[Allactaga]

Hi, @ZeroIntensity, this is definitely an existing issue unveiled. While there are a number of PRs addressing the "singleton issue", the root cause of #142414 is the redefinition of objects when importlib.reload is called. Not only UNBOUND is redefined, but so is QueueEmpty. Identity gets broken and as a result QueueEmpty is not recognised by except clauses where it is expected to.

Regardless what path forward is chosen for the refactoring of UnboundItem, we still need to merge #144096.

[Allactaga]

this is definitely an existing issue unveiled

Having said that, I do believe that the following change should not be merged:

exceptqueues.QueueEmpty:
pass

[github-actions]

This PR is stale because it has been open for 30 days with no activity.