Make pyvenv style virtual environments easier to configure when embedding Python

[grahamd]

BPO	22213
Nosy	@ncoghlan, @pitrou, @vstinner, @methane, @ericsnowcurrently, @zooba, @ndjensen, @LeslieGerman, @M-Kerr, @abrunner73
Dependencies	bpo-22257: PEP 432 (PEP 587): Redesign the interpreter startup sequence

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-08-17.11:42:41.318>
labels = ['type-feature', '3.8']
title = 'Make pyvenv style virtual environments easier to configure when embedding Python'
updated_at = <Date 2021-12-08.21:26:34.716>
user = 'https://bugs.python.org/grahamd'

bugs.python.org fields:

activity = <Date 2021-12-08.21:26:34.716>
actor = 'ndjensen'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2014-08-17.11:42:41.318>
creator = 'grahamd'
dependencies = ['22257']
files = []
hgrepos = []
issue_num = 22213
keywords = []
message_count = 31.0
messages = ['225434', '225436', '225437', '225739', '225742', '225771', '225774', '225890', '334926', '334948', '335015', '335468', '335470', '335479', '335484', '335648', '335650', '335688', '335692', '335749', '336793', '343636', '352905', '354856', '354857', '354858', '361600', '361869', '362260', '366570', '384496']
nosy_count = 13.0
nosy_names = ['ncoghlan', 'pitrou', 'vstinner', 'pyscripter', 'grahamd', 'methane', 'eric.snow', 'steve.dower', 'Henning.von.Bargen', 'ndjensen', 'Leslie', 'M.Kerr', 'abrunner73']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22213'
versions = ['Python 3.8']

Linked PRs

• GH-66409: check if exec_prefix is the same as prefix before searching executable_dir #127974

[grahamd]

In am embedded system, as the 'python' executable is itself not run and the Python interpreter is initialised in process explicitly using PyInitialize(), in order to find the location of the Python installation, an elaborate sequence of checks is run as implemented in calculate_path() of Modules/getpath.c.

The primary mechanism is usually to search for a 'python' executable on PATH and use that as a starting point. From that it then back tracks up the file system from the bin directory to arrive at what would be the perceived equivalent of PYTHONHOME. The lib/pythonX.Y directory under that for the matching version X.Y of Python being initialised would then be used.

Problems can often occur with the way this search is done though.

For example, if someone is not using the system Python installation but has installed a different version of Python under /usr/local. At run time, the correct Python shared library would be getting loaded from /usr/local/lib, but because the 'python' executable is found from /usr/bin, it uses /usr as sys.prefix instead of /usr/local.

This can cause two distinct problems.

The first is that there is no Python installation at all under /usr corresponding to the Python version which was embedded, with the result of it not being able to import 'site' module and therefore failing.

The second is that there is a Python installation of the same major/minor but potentially a different patch revision, or compiled with different binary API flags or different Unicode character width. The Python interpreter in this case may well be able to start up, but the mismatch in the Python modules or extension modules and the core Python library that was actually linked can cause odd errors or crashes to occur.

Anyway, that is the background.

For an embedded system the way this problem was overcome was for it to use Py_SetPythonHome() to forcibly override what should be used for PYTHONHOME so that the correct installation was found and used at runtime.

Now this would work quite happily even for Python virtual environments constructed using 'virtualenv' allowing the embedded system to be run in that separate virtual environment distinct from the main Python installation it was created from.

Although this works for Python virtual environments created using 'virtualenv', it doesn't work if the virtual environment was created using pyvenv.

One can easily illustrate the problem without even using an embedded system.

$ which python3.4
/Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4

$ pyvenv-3.4 py34-pyvenv

$ py34-pyvenv/bin/python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/private/tmp/py34-pyvenv'
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python34.zip', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plat-darwin', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload', '/private/tmp/py34-pyvenv/lib/python3.4/site-packages']

$ PYTHONHOME=/tmp/py34-pyvenv python3.4
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
Abort trap: 6

The basic problem is that in a pyvenv virtual environment, there is no duplication of stuff in lib/pythonX.Y, with the only thing in there being the site-packages directory.

When you start up the 'python' executable direct from the pyvenv virtual environment, the startup sequence checks know this and consult the pyvenv.cfg to extract the:

home = /Library/Frameworks/Python.framework/Versions/3.4/bin

setting and from that derive where the actual run time files are.

When PYTHONHOME or Py_SetPythonHome() is used, then the getpath.c checks blindly believe that is the authoritative value:

• Step 2. See if the $PYTHONHOME environment variable points to the
• installed location of the Python libraries. If $PYTHONHOME is set, then
• it points to prefix and exec_prefix. $PYTHONHOME can be a single
• directory, which is used for both, or the prefix and exec_prefix
• directories separated by a colon.

 /* If PYTHONHOME is set, we believe it unconditionally */
 if (home) {
 wchar_t *delim;
 wcsncpy(prefix, home, MAXPATHLEN);
 prefix[MAXPATHLEN] = L'\0';
 delim = wcschr(prefix, DELIM);
 if (delim)
 *delim = L'\0';
 joinpath(prefix, lib_python);
 joinpath(prefix, LANDMARK);
 return 1;
 }
Because of this, the problem above occurs as the proper runtime directories for files aren't included in sys.path. The result being that the 'encodings' module cannot even be found.

What I believe should occur is that PYTHONHOME should not be believed unconditionally. Instead there should be a check to see if that directory contains a pyvenv.cfg file and if there is one, realise it is a pyvenv style virtual environment and do the same sort of adjustments which would be made based on looking at what that pyvenv.cfg file contains.

For the record this issue is affecting Apache/mod_wsgi and right now the only workaround I have is to tell people that in addition to setting the configuration setting corresponding to PYTHONHOME, to use configuration settings to have the same effect as doing:

PYTHONPATH=/Library/Frameworks/Python.framework/Versions/3.4/lib/python34.zip:/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4:/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plat-darwin:/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload

so that the correct runtime files are found.

I am still trying to work out a more permanent workaround I can add to mod_wsgi code itself since can't rely on a fix for existing Python versions with pyvenv support.

Only other option is to tell people not to use pyvenv and use virtualenv instead.

Right now I can offer no actual patch as that getpath.c code is scary enough that not even sure at this point where the check should be incorporated or how.

Only thing I can surmise is that the current check for pyvenv.cfg being before the search for the prefix is meaning that it isn't consulted.

/* Search for an environment configuration file, first in the
 executable's directory and then in the parent directory.
 If found, open it for use when searching for prefixes.
*/

{
 wchar_t tmpbuffer[MAXPATHLEN+1];
 wchar_t *env_cfg = L"pyvenv.cfg";
 FILE * env_file = NULL;

 wcscpy(tmpbuffer, argv0_path);

 joinpath(tmpbuffer, env_cfg);
 env_file = _Py_wfopen(tmpbuffer, L"r");
 if (env_file == NULL) {
 errno = 0;
 reduce(tmpbuffer);
 reduce(tmpbuffer);
 joinpath(tmpbuffer, env_cfg);
 env_file = _Py_wfopen(tmpbuffer, L"r");
 if (env_file == NULL) {
 errno = 0;
 }
 }
 if (env_file != NULL) {
 /* Look for a 'home' variable and set argv0_path to it, if found */
 if (find_env_config_value(env_file, L"home", tmpbuffer)) {
 wcscpy(argv0_path, tmpbuffer);
 }
 fclose(env_file);
 env_file = NULL;
 }
 }

 pfound = search_for_prefix(argv0_path, home, _prefix, lib_python);

[ncoghlan]

Yeah, PEP-432 (my proposal to redesign the startup sequence) could just as well be subtitled "getpath.c hurts my brain" :P

One tricky part here is going to be figuring out how to test this - perhaps adding a new test option to _testembed and then running it both inside and outside a venv.

[ncoghlan]

Graham pointed out that setting PYTHONHOME ends up triggering the same control flow through getpath.c as calling Py_SetPythonHome, so this can be tested just with pyvenv and a suitably configured environment.

It may still be a little tricky though, since we normally run the pyvenv tests in isolated mode to avoid spurious failures due to bad environment settings...

[ncoghlan]

Some more experiments, comparing an installed vs uninstalled Python. One failure mode is that setting PYTHONHOME just plain breaks running from a source checkout (setting PYTHONHOME to the checkout directory also fails):

$ ./python -m venv --without-pip /tmp/issue22213-py35

$ /tmp/issue22213-py35/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
/usr/local /usr/local

$ PYTHONHOME=/usr/local /tmp/issue22213-py35/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
Aborted (core dumped)

Trying after running "make altinstall" (which I had previously done for 3.4) is a bit more enlightening:

$ python3.4 -m venv --without-pip /tmp/issue22213-py34

$ /tmp/issue22213-py34/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
/usr/local /usr/local

$ PYTHONHOME=/usr/local /tmp/issue22213-py34/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
/usr/local /usr/local

$ PYTHONHOME=/tmp/issue22213-py34 /tmp/issue22213-py34/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
Aborted (core dumped)

$ PYTHONHOME=/tmp/issue22213-py34:/usr/local /tmp/issue22213-py34/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
Aborted (core dumped)
[ncoghlan@lancre py34]$ PYTHONHOME=/usr/local:/tmp/issue22213-py34/bin /tmp/issue22213-py34/bin/python -c "import sys; print(sys.base_prefix, sys.base_exec_prefix)"
/usr/local /tmp/issue22213-py34/bin

I think what this is actually showing is that there's a fundamental conflict between mod_wsgi's expectation of being able to set PYTHONHOME to point to the virtual environment, and the way PEP-405 virtual environments actually work.

With PEP-405, all the operations in getpath.c expect to execute while pointing to the *base* environment: where the standard library lives. It is then up to site.py to later adjust the based prefix location, as can be demonstrated by the fact pyvenv.cfg isn't processed if processing the site module is disabled:

$ /tmp/issue22213-py34/bin/python -c "import sys; print(sys.prefix, sys.exec_prefix)"
/tmp/issue22213-py34 /tmp/issue22213-py34
$ /tmp/issue22213-py34/bin/python -S -c "import sys; print(sys.prefix, sys.exec_prefix)"
/usr/local /usr/local

At this point in time, there isn't an easy way for an embedding application to say "here's the standard library, here's the virtual environment with user packages" - it's necessary to just override the path calculations entirely.

Allowing that kind of more granular configuration is one of the design goals of PEP-432, so adding that as a dependency here.

[grahamd]

It is actually very easy for me to work around and I released a new mod_wsgi version today which works.

When I get a Python home option, instead of calling Py_SetPythonHome() with it, I append '/bin/python' to it and call Py_SetProgramName() instead.

[ncoghlan]

Excellent! If I recall correctly, that works because we resolve the symlink when looking for the standard library, but not when looking for venv configuration file.

I also suspect this is all thoroughly broken on Windows - there are so many configuration operations and platform specific considerations that need to be accounted for in getpath.c these days that it has become close to incomprehensible :(

One of my main goals with PEP-432 is actually to make it possible to rewrite the path configuration code in a more maintainable way - my unofficial subtitle for that PEP is "getpath.c must die!" :)

[grahamd]

I only make the change to Py_SetProgramName() on UNIX and not Windows. This is because back in mod_wsgi 1.0 I did actually used to use Py_SetProgramName() but it didn't seem to work in sane way on Windows so changed to Py_SetPythonHome() which worked on both Windows and UNIX. Latest versions of mod_wsgi haven't been updated yet to even build on Windows, so not caring about Windows right now.

[pitrou]

That workaround would definitely deserve being wrapped in a higher-level API invokable by embedding applications, IMHO.

[ncoghlan]

(Added Victor, Eric, and Steve to the nosy list here, as I'd actually forgotten about this until issue bpo-35706 reminded me)

Core of the problem: the embedding APIs don't currently offer a Windows-compatible way of setting up "use this base Python and this venv site-packages", and the way of getting it to work on other platforms is pretty obscure.

[zooba]

Victor may be thinking about it from time to time (or perhaps it's time to make the rest of the configuration changes plans concrete so we can all help out?), but I'd like to see this as either:

• a helper function to fill out the core config structure from a pyvenv.cfg file (rather than hiding it deeper as it currently is), or better yet,
• remove the dependency on all non-frozen imports at initialization and let embedders define Python code to do the initialization

In the latter case, the main python.exe also gets to define its behavior. So for the most part, we should be able to remove getpath[p].c and move it into the site module, then make that our Python initialization step.

This would also mean that if you are embedding Python but not allowing imports (e.g. as only a calculation engine), you don't have to do the dance of _denying_ all lookups, you simply don't initialize them.

But as far as I know, we don't have a concrete vision for "how will consumers embed Python in their apps" that can translate into work - we're still all individually pulling in slightly different directions. Sorting that out is most important - having someone willing to do the customer engagement work to define an actual set of requirements and desirables would be fantastic.

[ncoghlan]

Yeah, I mainly cc'ed Victor and Eric since making this easier ties into one of the original design goals for PEP-432 (even though I haven't managed to persuade either of them to become co-authors of that PEP yet).

[vstinner]

PEP-432 will allow to give with fine control on parameters used to initialize Python. Sadly, I failed to agree with Nick Coghlan and Eric Snow on the API. The current implementation (_PyCoreConfig and _PyMainInterpreterConfig) has some flaw (don't separate clearly the early initialization and Unicode-ready state, the interpreter contains main and core config whereas some options are duplicated in both configs, etc.).

See also bpo-35706.