Generated executable aliases do not work for packages that rely on multiprocessing (vermin)

[n67-exe]

Install source and version

• Installed from the Windows Store
• Installed with the MSIX from python.org
• Installed with the MSI from python.org
• Installed with winget install 9NQ7512CXL7T

Version: 26.2

Describe the bug
I encountered the bug while trying to use vermin (package) through the executable aliases that pymanager generates in %USERPROFILE%\AppData\Local\Python\bin. However i believe this is a much broader problem related to the use of multiprocessing module.
When multiprocessing is used with the spawn and forkserver start methods, child processes must be able to safely import the main module without executing the same threading logic.
For example, vermin doesn't execute its main function, it is called explicitly by an executable, making the main module (vermin.main) safe to import (as far as i understand). However, from my experimentation, the executable alias that pymanager provides (vermin.exe) calls vermin.exe.__script__.py, which isn't safe to import as it calls vermin.main() unconditionally.
I expect executable aliases to fail in a similar manner with other packages that rely on multiprocessing module the same way. But I wasn't able to confirm it yet.

I can think of a few ways to fix this issue:

• Calling the original executable directly if it exists in %USERPROFILE%\AppData\Local\Python\pythoncore-*\Scripts
  Probably the most reliable, but not very versatile.
• Making *.__script__.py files safe to import by adding an explicit __name__ check:

  if __name__ == '__main__':
   from ... import main
   sys.exit(main())

To Reproduce
Steps to reproduce the behavior (an example using vermin):

1. Install vermin 1.8.0

   py -m pip install vermin==1.8.0

2. Refresh executable aliases in %USERPROFILE%\AppData\Local\Python\bin

   py install --refresh

3. Ensure %USERPROFILE%\AppData\Local\Python\bin is in the PATH and %USERPROFILE%\AppData\Local\Python\pythoncore-*\Scripts is not (or has lower priority). Alternatively you can use %USERPROFILE%\AppData\Local\Python\bin\vermin in step 5 to emulate the same conditions.

   where vermin

4. Create an empty directory with a single empty main.py file.
5. From this directory run

   vermin --processes=2 .

6. See vermin infinitely calling itself and displaying errors

   RuntimeError: If running `detect_paths()` outside of `if __name__ == "__main__":`
   it, or the code calling it, must be done within it instead.
   No files specified to analyze!
   

Expected behavior
If you run vermin with --processes=1 (it avoids using multithreading in that case), or if you run the original vermin executable in %USERPROFILE%\AppData\Local\Python\pythoncore-*\Scripts instead, it will display the expected report without any errors:

Tips:
- Since '# novm' or '# novermin' weren't used, a speedup can be achieved using: --no-parse-comments
(disable using: --no-tips)

Minimum required versions: ~2, ~3

Note: Not enough evidence to conclude it won't work with Python 2 or 3.

Additional context
vermin.exe.__script__.py, generated consistently with py install --refresh.

[zooba]

I suspect something else is at play here, since vermin.exe.__script__.py isn't importable and shouldn't appear in sys.argv, but possibly multiprocessing has ways to find its way back to it.

Could you try adding if __name__ == '__main__': over the entire script file and see if that works? (It's fine to edit the script in place and then run the executable to launch it.) We shouldn't need to do anything at all if we're not actually launching it ourselves, and excluding the entire script is safest. I worry that it'll still miss some initialization that's needed for the app, though, so rather than debugging that on the next release it'd be good to flush out those problems now.

[n67-exe]

Could you try adding if __name__ == '__main__': over the entire script file and see if that works?

That works for vermin. In this case it is actually identical to what i suggested since vermin never uses sys.argv[0] apart from printing help which is always done in the main thread.
There is probably no difference since sys.argv in the child processes seems to be preserved from the main thread according to my experiments.

I suspect something else is at play here, since vermin.exe.__script__.py isn't importable and shouldn't appear in sys.argv

I'm not sure i understand what you meant by that.
All Windows systems use spawn start method by default (vermin doesn't change that). It involves child processes initiating a new python interpreter and then importing the __main__ module.
A quick peek into implementation of multiprocessing shows that it relies on sys.modules['__main__']:

# Alias for main module -- will be reset by bootstrapping child processes

if '__main__' in sys.modules:
 sys.modules['__mp_main__'] = sys.modules['__main__']

I added logging to both vermin.exe.__script__.py and vermin\main.py (to the beginning of the file) to see when they are imported:

print("__name__ =", __name__)
print("__file__ =", __file__)
print("argv =", sys.argv)
print("main module =", sys.modules["__main__"])
print("process id =", os.getpid())
print()

Running vermin --processes=1 . (through the alias in %USERPROFILE%\AppData\Local\Python\bin without any modifications) i get:

Output

__name__ = __main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=1', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 38340

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=1', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 38340

This is followed by vermin output. As expected each module is imported once, in order.

Running the broken vermin --processes=2 . (through the same alias) i get:

Output

__name__ = __main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 73556

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 73556

__name__ = __mp_main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 20344

__name__ = __mp_main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 71484

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 20344

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 71484

The last 4 entries repeat infinitely with different process ids interleaved with each other and errors. This shows that vermin.exe.__script__.py is imported from the child threads as well. Since it calls vermin.main() when imported this leads to child processes creating their own child processes and so on.

With the added if __name__ == '__main__': to vermin.exe.__script__.py i get the following output:

Output

__name__ = __main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 59684

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 59684

__name__ = __mp_main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 162640

__name__ = __mp_main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 136740

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__mp_main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 162640

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__mp_main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 136740

__name__ = __mp_main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 80904

__name__ = __mp_main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 129492

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__mp_main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 80904

This is again followed by vermin output. There is no difference in the output if i add if __name__ == '__main__': over the entire script instead of just the main call. The thread pool is created twice in the vermin implementation, so we see 2 pairs of threads launched one after another. Depending on the timing and the amount of work available, one of the threads might not perform any work (like the thread 129492 in this run).

[zooba]

Ah, I see. We're not updating the module's __spec__.origin (aka __file__), and that's what is being used to get back to the script file.

To be clear, this is entirely multiprocessing's doing and not Vermin - I expect any generic multiprocessing script would behave the same (I just haven't had time to build one to test it).

Unfortunately, I expect that running a __main__ that does nothing is also likely to break multiprocessing-based apps that expect it to do something. There's not really much we can do about that.

Another snippet that might work without the __name__ == '__main__' check is this one (anywhere in the file before the main call). Could you try putting that at the top of the script file, remove the other check we added, and see if that also works?

__file__ = None
try:
 __spec__.origin = None
except Exception:
 pass

(I prefer this one because nobody should ever import these script files, and the sooner they break because they are trying, the easier it is to figure out what they're getting wrong. If it looks like it works, the eventual error could be far removed from the actual mistake, which means it's much harder to debug the issue.)

[n67-exe]

Your suggested approach works with vermin and it prevents the script from being imported at all in child threads.
With the following vermin.exe.__script__.py (added more logging before and after checks):

Script

import sys
import os

print("Launcher script before checks:")
print("__name__ =", __name__)
print("__file__ =", __file__)
print("argv =", sys.argv)
print("main module =", sys.modules["__main__"])
print("process id =", os.getpid())
print()

__file__ = None
try:
 __spec__.origin = None
except Exception:
 pass

import sys

try:
 if not sys.path[0]:
 del sys.path[0]
except AttributeError:
 pass
except IndexError:
 pass

# Replace argv[0] with our executable instead of the script name.
try:
 if sys.argv[0][-14:].upper() == ".__SCRIPT__.PY":
 sys.argv[0] = sys.argv[0][:-14]
 sys.orig_argv[0] = sys.argv[0]
except AttributeError:
 pass
except IndexError:
 pass

print("Launcher script after checks:")
print("__name__ =", __name__)
print("__file__ =", __file__)
print("argv =", sys.argv)
print("main module =", sys.modules["__main__"])
print("process id =", os.getpid())
print()

from vermin import main
sys.exit(main())

I get this output for vermin --processes=1 .:

Output

Launcher script before checks:
__name__ = __main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=1', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 29352

Launcher script after checks:
__name__ = __main__
__file__ = None
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=1', '.']
main module = <module '__main__' from None>
process id = 29352

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=1', '.']
main module = <module '__main__' from None>
process id = 29352

I get this output for vermin --processes=2 .:

Output

Launcher script before checks:
__name__ = __main__
__file__ = %USERPROFILE%\AppData\Local\Python\bin\vermin.exe.__script__.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py', '--processes=2', '.']
main module = <module '__main__' from '%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe.__script__.py'>
process id = 102056

Launcher script after checks:
__name__ = __main__
__file__ = None
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' from None>
process id = 102056

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' from None>
process id = 102056

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 32336

__name__ = vermin.main
__file__ = %USERPROFILE%\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\vermin\main.py
argv = ['%USERPROFILE%\\AppData\\Local\\Python\\bin\\vermin.exe', '--processes=2', '.']
main module = <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
process id = 154204

The difference compared to the if __name__ == '__main__': fix is that child threads never import vermin.exe.__script__.py, and as a side effect module __mp_main__ never appears in the log. While __mp_main__ is a valid entry in sys.modules for child threads, in this case it has the name __main__ (it always equals main in vermin.main module). I have no idea if this is a problem and if any package relies on __mp_main__ in some way.

Another point to consider is that several launcher scripts generated by other libraries (distlib, pip, uv) use a simple if __name__ == '__main__': check.
My personal opinion is that it would be reasonable to be consistent with other implementations (to avoid any "pymanger specific" problems). That or just relying on already existing executables generated in %USERPROFILE%\AppData\Local\Python\pythoncore-*\Scripts as i suggested earlier.

[zooba]

it would be reasonable to be consistent with other implementations

This has some validity, though it is strictly more correct for the script to not be loaded at all. Anything relying on __mp_main__ is almost certainly making an invalid assumption, and it's definitely reliant on private/undocumented details. I think I'll go with both.

Thanks for the help testing.

relying on already existing executables generated ...

That's fine if you want to add it to your PATH instead of the PyManager directory. But we aren't able to manage those for you, and can't add those directories for you, so you're on your own.

Our official recommendation is to create a virtual environment and activate it to get the correct scripts on PATH.

[n67-exe]

That's fine if you want to add it to your PATH instead of the PyManager directory. But we aren't able to manage those for you, and can't add those directories for you, so you're on your own.

Our official recommendation is to create a virtual environment and activate it to get the correct scripts on PATH.

What i meant is that inside %USERPROFILE%\AppData\Local\Python\bin a generated executable alias (.exe) can use the target interpreter path (from .exe.__target__) to locate %USERPROFILE%\AppData\Local\Python\pythoncore-*\Scripts directory and launch an executable directly from there. This way it bypasses a python launcher script (.exe.__script__.py) and thus avoids the problem of maintaining such script entirely on our part.
This of course assumes that for every alias pymanager creates there already exists an executable in %USERPROFILE%\AppData\Local\Python\pythoncore-*\Scripts which sounds like it's always true to me but i don't know for sure.

[zooba]

sounds like it's always true to me but i don't know for sure

There are two totally independent tools involved, so not necessarily, though it's reasonable that if someone has tried to disable pip from generating their aliases (which is hard), they probably don't care to have PyManager's aliases either (though disabling them is easy). So in practice, most of the time pip will generate aliases. But with literal millions of users, even 99% isn't enough to stop thousands of people being broken. So we can't rely on it.

Plus there's the performance cost of multiple executables being involved. Our aliases are written specifically to avoid too many processes being created, but distlib's aren't (and probably can't be), so launching a PyManager alias for a recent CPython should be noticeably faster (and will be more reliable if you're trying to wait for it to complete).

The tradeoff is there may be slightly different behaviour, at least until we smooth out the bumps (the PyManager support has been there less than a year for aliases), but it's worth it for the better/safer/faster result.