Make python slightly more relocatable

[mathias]

BPO	18309
Nosy	@ronaldoussoren, @ncoghlan, @ericsnowcurrently, @shakfu
Files	python-relative-path-lookup.diff: Proposed change python-relative-path-lookup-v2.diff workaround.c: example of workaround to obtain python home in a relocatable plugin scenario.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-06-26.16:00:49.967>
labels = ['interpreter-core', '3.9']
title = 'Make python slightly more relocatable'
updated_at = <Date 2022-03-18.10:35:06.782>
user = 'https://bugs.python.org/mathias'

bugs.python.org fields:

activity = <Date 2022-03-18.10:35:06.782>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2013-06-26.16:00:49.967>
creator = 'mathias'
dependencies = []
files = ['30706', '30768', '50677']
hgrepos = []
issue_num = 18309
keywords = ['patch']
message_count = 17.0
messages = ['191909', '191929', '191935', '191944', '191994', '192277', '353243', '353267', '353276', '353288', '353311', '353720', '415142', '415145', '415148', '415172', '415218']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'ncoghlan', 'eric.snow', 'mathias', 'shakfu']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue18309'
versions = ['Python 3.9']

[mathias]

Hi all,

I want to move python a bit closer to be relocatable.
One problem to solve is where python finds its modules.
The usual lookup mechanism is to compile in a configure time
determined prefix that is used as a last resort path if the
paths are not set otherwise during application/interpreter startup.
The most commonly known way to change the module path at startup time
are probably the environment variables PYTHONPATH and PYTHONHOME.
The python interpreter itself already tries to interpret argv[0] to get to this point, but it would be nice if an application embedded interpreter also finds its module path without providing this argv[0] directly to the python library. This should even work if being moved or being installed at a different path than the configure time prefix path.

The proposal is to add an additional attempt to find the python modules
just before we resort to the compiled in prefix by looking at the path
to the python27.{so,dll}. Relative to this shared object python library
file the python modules are searched in the usual way. If there are
no python modules found relative to the python library file, the very
last resort compiled in prefix is used as usual.

For architectures where we cannot determine the path of the shared
library file, nothing changes.

I have attached a patch that tries to implement this.
It should serve as a base for discussions.
This change is tested on linux and behaves like expected. The windows code for this is copied over from an other project where I have this actively running. But this python code variant is not even compile tested on windows.

thanks in advance

Mathias

[ericsnowcurrently]

Hi Mathias. There is a current proposal (http://www.python.org/dev/peps/pep-0432/) for improving interpreter startup. So changes in this area are subject to extra caution. The changes you are talking about are at least indirectly impacted by the proposal, though I expect they are more directly tied to what happens in site.py.

As to your proposal, aren't the embedding needs already addressed? See http://docs.python.org/2/c-api/intro.html#embedding-python. Is there some convention for keeping the site files adjacent to the SO/DLL that would warrant your proposed code?

p.s. this would be a new feature so it only applies to Python 3.4.

[mathias]

Hi Eric,

Thanks for looking at that ticket so fast!

Reassigning this to 3.4 is great.

In general, yes I can already do what I need more or less. This is the reason why I can be fine with about every python version.

The point I bring up this change that I believe I am doing this at an unappropriate place as I need to know some internals of python when I do so and that I think that other can probably also benefit from this idea/change.
What I currently do is to write an application that just uses python*.so as an embedded interpreter and this precompiled application might be relocated to about everywhere - just where it is unpacked.
We are currently using the same sort of code to find out where the python*so file is and we use Py_SetPythonHome to set is to the directory where the so file resides.

Why are we doing this?
So, it takes the idea that is currently in the standard python interpreter. This one tries to be relocatable (means: pack the installation directory and unpack that somewhere else and be still able to run) by looking at argv[0] and dereferencing symbolic links until it arrives at a real file.
Now suppose you want to embed python, then you do no longer use the standard python interpreter program. You may also use a different installation layout for basic things like bin and lib. So you end up with an application that is no longer able to find its provided python modules by looking at the applications path.
But instead of starting from the path of the interpreter (which is not used in this case) or the application itself you could start from the python library path and look for your python installation relative to that. So as long as you stick with the relative file layout of everything that is python related (and only what is python related, the python.so and the modules) when you pack and unpack your precompiled application this would just work.

So, put that in short:
Instead of dynamically finding the the python module path relative to .../bin/python try to find the python relative to .../lib/libpython34.so.
The benefit of that would be that every application that embeds python and needs to be relocatable will just work in the way that today only the standard python interpreter works.

I try to get all of the PEP you pointed me to.
As I am seeing this longer document the first time, I am not sure if I missed something there, but in that framework of this my proposal would probably influence the initial setting of

sys.prefix (?)

if this is not already provided from the embedding application.

And yes I am perfectly fine with a different or more general approach.
The initially attached patch is something that tried to integrate into the current checked in code as I understood it.

Greetings

Mathias

[ncoghlan]

The way we figure out where to find the standard library is crazy, and creating the infrastructure to start making it less crazy is actually one of the prime motivations for PEP-432 :)

[ronaldoussoren]

Note that the OSX port already does this for framework builds. I don't know why we don't use the same code for shared library builds.

Issue bpo-15498 contains a patch that switches this code from a deprecated nextstep-era API to dladdr.

Two comments on the patch attached to this issue:

1. The name "_PyImport_GetModulePath" is confusing, I'd use _PyImport_GetSharedLibPath to make clear that this is locating the shared library.

2. The code calls dladdr on a static variable that's introduced just for that, it is also possible to call dladdr on an already existing symbol (for example the address of a function in the public API).

[mathias]

Hi Ronald, Eric, Nick,

Looking up the symbol name of the current function should work also.
And I am free to rename these functions to whatever you like.

Attached is version 2 of the patch with the suggested changes.
The windows implementation is still untested.

It would be interesting to know if this kind of lookup scheme can be included into PEP-432.
Provided the spirit of this PEP, I can imagine to provide several functions to build up the pythonpath starting from something.
So say, have a 'get python path from argv[0]', a 'get python path from shared python library' and a 'get python path from prefix' function (I may miss a variant). Also a 'build python path from python home root entry point' function would be useful and could be used by the above functions.
An application embedding python can then call those functions that are sensible for its own use and installation scheme to set the module path in the PyConfig struct. The Py_ReadConfig function will internally use the above suggested functions to build up the default configuration if not already provided.

Greetings

Mathias

[vstinner]

The PEP-587 "Python Initialization Configuration" has been implemented in Python 3.8. It provides fine control on the "Path Configuration":

• https://docs.python.org/dev/c-api/init_config.html
• https://docs.python.org/dev/c-api/init_config.html#init-path-config

[mathias]

Hi,

Nice to see some progress.
Still, I checked todays https://github.com/python/cpython.git master and 3.8 branch (is that the current cpython development code?). Neither of them contain a call to dladdr beside the macos code path mentioned in msg191994 by Ronald Oussoren which does this already for a long time.
By the lack of dladdr, I conclude that the code idea of my request here is not solved.

May be to rephrase that. The basic idea behind that request was to make
pythons default way to setup the paths required to find the python modules based on the place where the python library resides instead of the python executable program. I do not mean the compile time prefix but the actual location of the shared object in the file system.
That would help to build applications that embed cpython, ship and unpack the whole application tree including the python modules to a custom location, while still preserving the subtree structure containing the python shared library and the python modules, not known at compile time. Note that this patch contained code to make that work from within python without custom code in the embedding application. Doing that on the embedding and calling application side was always possible and still is possible - but that was not the point.

best

Mathias

[vstinner]

Hum, I am confused. I understood that this issue is able customizing sys.path when Python is embedded. But it seems like the feature request is more about the *default* implementation, not how to reimplement it outside Python (with custm code).