gh-134939: Add a Multiple Interpreters Howto Doc by ericsnowcurrently · Pull Request #136143 · python/cpython

ericsnowcurrently · 2025-06-30T17:07:38Z

Issue: Add the interpreters Module, Implementing PEP 734 #134939

📚 Documentation preview 📚: https://cpython-previews--136143.org.readthedocs.build/en/136143/howto/multiple-interpreters.html#multiple-interpreters-howto

Co-authored-by: neonene <53406459+neonene@users.noreply.github.com>

neonene · 2025-07-02T17:39:08Z

+ are still relatively inefficient and limited
+* actually *sharing* data safely is tricky (true for free-threading too)
+* all necessary modules must be imported separately in each interpreter
+* relatively slow startup time per interpreter


Some users might need a (foot)note that subinterpreters can sometimes improve the startup by importing nothing.
example
defnon_stateless(): non_statelessif__name__=='__main__': importthreadingfromconcurrentimportinterpretersinterps= [interpreters.create() foriinrange(10)] threads= [ threading.Thread(target=interp.call, args=(non_stateless,)) forinterpininterps ] forthreadinthreads: thread.start() forthreadinthreads: thread.join()
That seems not so beneficial for the InterpreterPoolExecutor case, though.

I've added a note, though further down in the discussion about calling functions.

neonene · 2025-08-01T05:37:58Z

+
+ if __name__ == '__main__':
+ interp = interpreters.create()
+ t = interp.call_in_thread(script)


Off-topic, but would it be possible to add the daemon parameter to the helper method?

I want to keep the helper super simple. If someone wants a daemon thread then they can do it manually without much effort.

ericsnowcurrently · 2025-09-25T23:42:19Z

Any objections to merging this?

hugovk · 2025-09-30T14:51:01Z

+ example = dedent(os.linesep.join(current))
+ expected = ''.join(f'{l}{os.linesep}' for l in expected)
+ examples[start] = (example, expected)
+ current = expected = start = None


This line is unused:
Suggested change
current = expected = start = None

hugovk · 2025-09-30T14:51:24Z

+def write_example(examplesdir, name, text):
+ filename = os.path.join(examplesdir, f'example-{name}.py')


Suggested change
defwrite_example(examplesdir, name, text):
filename=os.path.join(examplesdir, f'example-{name}.py')
defwrite_example(examples_dir, name, text):
filename=os.path.join(examples_dir, f'example-{name}.py')

hugovk · 2025-09-30T15:00:35Z

+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,


Suggested change
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
capture_output=True,

hugovk · 2025-09-30T15:00:53Z

+ if failed:
+ print(f'{len(failed)} failed: {",".join(failed)}')
+ else:
+ print(f'all succeeded')


Suggested change
print(f'all succeeded')
print('all succeeded')

hugovk · 2025-10-01T13:12:48Z

 name: Run Ruff (lint) on Doc/
 args: [--exit-non-zero-on-fix]
 files: ^Doc/
+ exclude: ^Doc/howto/multiple-interpreters-run-examples.py


Let's remove these exclusions and do the fixes?
Suggested change
exclude: ^Doc/howto/multiple-interpreters-run-examples.py

hugovk · 2025-10-01T13:29:24Z

+Sharing Data
+------------


Suggested change
Sharing Data
------------
Sharing data
------------

hugovk · 2025-10-01T13:29:48Z

+Using Queues
+------------


Suggested change
Using Queues
------------
Using queues
------------

hugovk · 2025-10-01T13:30:07Z

+Initializing Globals for a Script
+---------------------------------


Suggested change
Initializing Globals for a Script
---------------------------------
Initializing globals for a script
---------------------------------

hugovk · 2025-10-01T13:31:33Z

+Pipes, Sockets, etc.
+--------------------


Suggested change
Pipes, Sockets, etc.
--------------------
Pipes, sockets, and so on
-------------------------
Avoid Latin abbreviations: https://devguide.python.org/documentation/style-guide/#use-simple-language

hugovk · 2025-10-01T13:33:11Z

+Tutorial: Miscellaneous
+=======================


Suggested change
Tutorial: Miscellaneous
=======================
Tutorial: miscellaneous
=======================

github-actions · 2026-04-26T12:14:45Z

This PR is stale because it has been open for 30 days with no activity.

ericsnowcurrently added 5 commits June 30, 2025 10:57

Add the multiple interpreters howto doc.
96e1e78

Drop extraneous changes.
e5e980e

Finish howto doc, minus recipes.
2fcaedd

Add a TODO list.
aea8278

Drop the execcomponents ref.
4bb1e2f

ericsnowcurrently added the needs backport to 3.14 bugs and security fixes label Jun 30, 2025

bedevere-appBot added docs Documentation in the Doc dir skip news labels Jun 30, 2025

github-project-automationBot added this to Docs PRs Jun 30, 2025

github-project-automationBot moved this to Todo in Docs PRs Jun 30, 2025

bedevere-appBot mentioned this pull request Jun 30, 2025
Add the interpreters Module, Implementing PEP 734 #134939
Open
4 tasks

ericsnowcurrently added 2 commits June 30, 2025 16:56

Fix a ref.
d1ab402

Drop the examples section for now.
1c2d40f

ericsnowcurrently marked this pull request as ready for review June 30, 2025 23:00

bedevere-appBot added the awaiting core review label Jun 30, 2025

neonene reviewed Jul 1, 2025
View reviewed changes

ericsnowcurrentlyand others added 6 commits July 1, 2025 10:55

Fix typos.
ade2ce3
Co-authored-by: neonene <53406459+neonene@users.noreply.github.com>

Add a misc section to the tutorial.
40f50e7

Clarify about prepare_main().
0c3105b

Fix a pseudo-ref.
6028349

Fix the examples.
d7a7cf0

Tweak one example.
fb13944

neonene reviewed Jul 2, 2025
View reviewed changes

ericsnowcurrently added 6 commits July 2, 2025 13:04

Add a ref.
72e46fb

Clarify about Interpreter.exec().
675246d

Clarify about calling different kinds of function.
22e3ee0

Clarify a note.
f7661da

Add a caveat about -c and the REPL.
28f0218

Add a pro tip.
1b79755

neonene reviewed Aug 1, 2025
View reviewed changes

ericsnowcurrently added 2 commits September 23, 2025 16:54

Add run-examples.py and fix some of the examples.
b75c380

Do not try reformatting run-examples.py.
35101a0

ericsnowcurrently requested a review from hugovk as a code owner September 24, 2025 19:00

ericsnowcurrently added 2 commits September 24, 2025 15:16

Fix some of the examples.
ddae34c

Do not lint run-examples.py.
652abd3

ericsnowcurrently mentioned this pull request Sep 30, 2025
python3.14.0rc3 idle doesn't show subinterpreter output while console does #139383
Open

hugovk reviewed Oct 1, 2025
View reviewed changes

github-actionsBot added the stale Stale PR or inactive for long period of time. label Apr 26, 2026

serhiy-storchaka added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label May 30, 2026

github-actionsBot removed the stale Stale PR or inactive for long period of time. label Jun 1, 2026

		def write_example(examplesdir, name, text):
		filename = os.path.join(examplesdir, f'example-{name}.py')

-defwrite_example(examplesdir, name, text):
-filename=os.path.join(examplesdir, f'example-{name}.py')
+defwrite_example(examples_dir, name, text):
+filename=os.path.join(examples_dir, f'example-{name}.py')

	stdout=subprocess.PIPE,
	stderr=subprocess.PIPE,
	capture_output=True,

		Initializing Globals for a Script
		---------------------------------

Uh oh!

Uh oh!

Conversation

ericsnowcurrently commented Jun 30, 2025• edited by hugovk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently commented Sep 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ericsnowcurrently commented Jun 30, 2025•
edited by hugovk
Loading