gh-134939: Add the interpreters Module

.github/CODEOWNERS

@@ -281,9 +281,13 @@ Doc/howto/clinic.rst @erlend-aasland
 # Subinterpreters
 **/*interpreteridobject.* @ericsnowcurrently
 **/*crossinterp* @ericsnowcurrently
-Lib/test/support/interpreters/ @ericsnowcurrently
 Modules/_interp*module.c @ericsnowcurrently
+Lib/test/test__interp*.py @ericsnowcurrently
+Lib/concurrent/interpreters/ @ericsnowcurrently
+Lib/test/support/channels.py @ericsnowcurrently
+Doc/library/concurrent.interpreters.rst @ericsnowcurrently
 Lib/test/test_interpreters/ @ericsnowcurrently
+Lib/concurrent/futures/interpreter.py @ericsnowcurrently
 
 # Android
 **/*Android* @mhsmith @freakboy3742

Doc/library/concurrency.rst

@@ -18,6 +18,7 @@ multitasking). Here's an overview:
  multiprocessing.shared_memory.rst
  concurrent.rst
  concurrent.futures.rst
+ concurrent.interpreters.rst
  subprocess.rst
  sched.rst
  queue.rst

Doc/library/concurrent.interpreters.rst

@@ -0,0 +1,198 @@
+:mod:`!concurrent.interpreters` --- Multiple interpreters in the same process
+=============================================================================
+
+.. module:: concurrent.interpreters
+ :synopsis: Multiple interpreters in the same process
+
+.. moduleauthor:: Eric Snow <ericsnowcurrently@gmail.com>
+.. sectionauthor:: Eric Snow <ericsnowcurrently@gmail.com>
+
+.. versionadded:: 3.14
+
+**Source code:** :source:`Lib/concurrent/interpreters.py`
+
+--------------
+
+
+Introduction
+------------
+
+The :mod:`!concurrent.interpreters` module constructs higher-level
+interfaces on top of the lower level :mod:`!_interpreters` module.
+
+.. XXX Add references to the upcoming HOWTO docs in the seealso block.
+
+.. seealso::
+
+ :ref:`isolating-extensions-howto`
+ how to update an extension module to support multiple interpreters
+
+ :pep:`554`
+
+ :pep:`734`
+
+ :pep:`684`
+
+.. XXX Why do we disallow multiple interpreters on WASM?
+
+.. include:: ../includes/wasm-notavail.rst
+
+
+Key details
+-----------
+
+Before we dive into examples, there are a small number of details
+to keep in mind about using multiple interpreters:
+
+* isolated, by default
+* no implicit threads
+* not all PyPI packages support use in multiple interpreters yet
+
+.. XXX Are there other relevant details to list?
+
+In the context of multiple interpreters, "isolated" means that
+different interpreters do not share any state. In practice, there is some
+process-global data they all share, but that is managed by the runtime.
+
+
+Reference
+---------
+
+This module defines the following functions:
+
+.. function:: list_all()
+
+ Return a :class:`list` of :class:`Interpreter` objects,
+ one for each existing interpreter.
+
+.. function:: get_current()
+
+ Return an :class:`Interpreter` object for the currently running
+ interpreter.
+
+.. function:: get_main()
+
+ Return an :class:`Interpreter` object for the main interpreter.
+
+.. function:: create()
+
+ Initialize a new (idle) Python interpreter
+ and return a :class:`Interpreter` object for it.
+
+
+Interpreter objects
+^^^^^^^^^^^^^^^^^^^
+
+.. class:: Interpreter(id)
+
+ A single interpreter in the current process.
+
+ Generally, :class:`Interpreter` shouldn't be called directly.
+ Instead, use :func:`create` or one of the other module functions.
+
+ .. attribute:: id
+
+ (read-only)
+
+ The interpreter's ID.
+
+ .. attribute:: whence
+
+ (read-only)
+
+ A string describing where the interpreter came from.
+
+ .. method:: is_running()
+
+ Return ``True`` if the interpreter is currently executing code
+ in its :mod:`!__main__` module and ``False`` otherwise.
+
+ .. method:: close()
+
+ Finalize and destroy the interpreter.
+
+ .. method:: prepare_main(ns=None, **kwargs)
+
+ Bind "shareable" objects in the interpreter's
+ :mod:`!__main__` module.
+
+ .. method:: exec(code, /, dedent=True)
+
+ Run the given source code in the interpreter (in the current thread).
+
+ .. method:: call(callable, /, *args, **kwargs)
+
+ Return the result of calling running the given function in the
+ interpreter (in the current thread).
+
+ .. method:: call_in_thread(callable, /, *args, **kwargs)
+
+ Run the given function in the interpreter (in a new thread).
+
+Exceptions
+^^^^^^^^^^
+
+.. exception:: InterpreterError
+
+ This exception, a subclass of :exc:`Exception`, is raised when
+ an interpreter-related error happens.
+
+.. exception:: InterpreterNotFoundError
+
+ This exception, a subclass of :exc:`InterpreterError`, is raised when
+ the targeted interpreter no longer exists.
+
+.. exception:: ExecutionFailed
+
+ This exception, a subclass of :exc:`InterpreterError`, is raised when
+ the running code raised an uncaught exception.
+
+ .. attribute:: excinfo
+
+ A basic snapshot of the exception raised in the other interpreter.
+
+.. XXX Document the excinfoattrs?
+
+.. exception:: NotShareableError
+
+ This exception, a subclass of :exc:`TypeError`, is raised when
+ an object cannot be sent to another interpreter.
+
+
+.. XXX Add functions for communicating between interpreters.
+
+
+Basic usage
+-----------
+
+Creating an interpreter and running code in it::
+
+ from concurrent import interpreters
+
+ interp = interpreters.create()
+
+ # Run in the current OS thread.
+
+ interp.exec('print("spam!")')
+
+ interp.exec("""if True:
+ print('spam!')
+ """)
+
+ from textwrap import dedent
+ interp.exec(dedent("""
+ print('spam!')
+ """))
+
+ def run():
+ print('spam!')
+
+ interp.call(run)
+
+ # Run in new OS thread.
+
+ t = interp.call_in_thread(run)
+ t.join()
+
+
+.. XXX Explain about object "sharing".

Doc/library/concurrent.rst

@@ -1,6 +1,7 @@
 The :mod:`!concurrent` package
 ==============================
 
-Currently, there is only one module in this package:
+This package contains the following modules:
 
 * :mod:`concurrent.futures` -- Launching parallel tasks
+* :mod:`concurrent.interpreters` -- Multiple interpreters in the same process

Doc/library/python.rst

@@ -27,3 +27,8 @@ overview:
  inspect.rst
  annotationlib.rst
  site.rst
+
+.. seealso::
+
+ * See the :mod:`concurrent.interpreters` module, which similarly
+ exposes core runtime functionality.

Doc/whatsnew/3.14.rst

@@ -83,6 +83,7 @@ and improvements in user-friendliness and correctness.
 .. PEP-sized items next.
 
 * :ref:`PEP 649 and 749: deferred evaluation of annotations <whatsnew314-pep649>`
+* :ref:`PEP 734: Multiple Interpreters in the Stdlib <whatsnew314-pep734>`
 * :ref:`PEP 741: Python Configuration C API <whatsnew314-pep741>`
 * :ref:`PEP 750: Template strings <whatsnew314-pep750>`
 * :ref:`PEP 758: Allow except and except* expressions without parentheses <whatsnew314-pep758>`
@@ -123,6 +124,101 @@ of Python. See :ref:`below <whatsnew314-refcount>` for details.
 New features
 ============
 
+.. _whatsnew314-pep734:
+
+PEP 734: Multiple Interpreters in the Stdlib
+--------------------------------------------
+
+The CPython runtime supports running multiple copies of Python in the
+same process simultaneously and has done so for over 20 years.
+Each of these separate copies is called an "interpreter".
+However, the feature had been available only through the C-API.
+
+That limitation is removed in the 3.14 release,
+with the new :mod:`concurrent.interpreters` module.
+
+There are at least two notable reasons why using multiple interpreters
+is worth considering:
+
+* they support a new (to Python), human-friendly concurrency model
+* true multi-core parallelism
+
+For some use cases, concurrency in software enables efficiency and
+can simplify software, at a high level. At the same time, implementing
+and maintaining all but the simplest concurrency is often a struggle
+for the human brain. That especially applies to plain threads
+(for example, :mod:`threading`), where all memory is shared between all threads.
+
+With multiple isolated interpreters, you can take advantage of a class
+of concurrency models, like CSP or the actor model, that have found
+success in other programming languages, like Smalltalk, Erlang,
+Haskell, and Go. Think of multiple interpreters like threads
+but with opt-in sharing.
+
+Regarding multi-core parallelism: as of the 3.12 release, interpreters
+are now sufficiently isolated from one another to be used in parallel.
+(See :pep:`684`.) This unlocks a variety of CPU-intensive use cases
+for Python that were limited by the :term:`GIL`.
+
+Using multiple interpreters is similar in many ways to
+:mod:`multiprocessing`, in that they both provide isolated logical
+"processes" that can run in parallel, with no sharing by default.
+However, when using multiple interpreters, an application will use
+fewer system resources and will operate more efficiently (since it
+stays within the same process). Think of multiple interpreters as
+having the isolation of processes with the efficiency of threads.
+
+.. XXX Add an example or two.
+.. XXX Link to the not-yet-added HOWTO doc.
+
+While the feature has been around for decades, multiple interpreters
+have not been used widely, due to low awareness and the lack of a stdlib
+module. Consequently, they currently have several notable limitations,
+which will improve significantly now that the feature is finally
+going mainstream.
+
+Current limitations:
+
+* starting each interpreter has not been optimized yet
+* each interpreter uses more memory than necessary
+ (we will be working next on extensive internal sharing between
+ interpreters)
+* there aren't many options *yet* for truly sharing objects or other
+ data between interpreters (other than :type:`memoryview`)
+* many extension modules on PyPI are not compatible with multiple
+ interpreters yet (stdlib extension modules *are* compatible)
+* the approach to writing applications that use multiple isolated
+ interpreters is mostly unfamiliar to Python users, for now
+
+The impact of these limitations will depend on future CPython
+improvements, how interpreters are used, and what the community solves
+through PyPI packages. Depending on the use case, the limitations may
+not have much impact, so try it out!
+
+Furthermore, future CPython releases will reduce or eliminate overhead
+and provide utilities that are less appropriate on PyPI. In the
+meantime, most of the limitations can also be addressed through
+extension modules, meaning PyPI packages can fill any gap for 3.14, and
+even back to 3.12 where interpreters were finally properly isolated and
+stopped sharing the :term:`GIL`. Likewise, we expect to slowly see
+libraries on PyPI for high-level abstractions on top of interpreters.
+
+Regarding extension modules, work is in progress to update some PyPI
+projects, as well as tools like Cython, pybind11, nanobind, and PyO3.
+The steps for isolating an extension module are found at
+:ref:`isolating-extensions-howto`. Isolating a module has a lot of
+overlap with what is required to support
+:ref:`free-threading <whatsnew314-free-threaded-cpython>`,
+so the ongoing work in the community in that area will help accelerate
+support for multiple interpreters.
+
+Also added in 3.14: :ref:`concurrent.futures.InterpreterPoolExecutor
+<whatsnew314-concurrent-futures-interp-pool>`.
+
+.. seealso::
+ :pep:`734`.
+
+
 .. _whatsnew314-pep750:
 
 PEP 750: Template strings
@@ -1108,6 +1204,8 @@ calendar
 concurrent.futures
 ------------------
 
+.. _whatsnew314-concurrent-futures-interp-pool:
+
 * Add :class:`~concurrent.futures.InterpreterPoolExecutor`,
  which exposes "subinterpreters" (multiple Python interpreters in the
  same process) to Python code. This is separate from the proposed API

Lib/concurrent/futures/interpreter.py

@@ -167,7 +167,7 @@ def run(self, task):
  except _interpqueues.QueueError:
  continue
  except ModuleNotFoundError:
- # interpreters.queues doesn't exist, which means
+ # interpreters._queues doesn't exist, which means
  # QueueEmpty doesn't. Act as though it does.
  continue
  else: