Portable python3 shebang for Windows, macOS, and Linux

[joshtriplett]

BPO	46645
Nosy	@lazka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-05.02:02:13.851>
labels = ['type-bug', '3.11']
title = 'Portable python3 shebang for Windows, macOS, and Linux'
updated_at = <Date 2022-02-05.08:20:36.092>
user = 'https://bugs.python.org/joshtriplett'

bugs.python.org fields:

activity = <Date 2022-02-05.08:20:36.092>
actor = 'lazka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2022-02-05.02:02:13.851>
creator = 'joshtriplett'
dependencies = []
files = []
hgrepos = []
issue_num = 46645
keywords = []
message_count = 3.0
messages = ['412553', '412555', '412563']
nosy_count = 2.0
nosy_names = ['joshtriplett', 'lazka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46645'
versions = ['Python 3.11']

[joshtriplett]

I'm writing this issue on behalf of the Rust project.

The build system for the Rust compiler is a Python 3 script x.py, which orchestrates the build process for a user even if they don't already have Rust installed. (For instance, x.py build, x.py test, and various command-line arguments for more complex cases.)

We currently run into various issues making this script easy for people to use on all common platforms people build Rust on: Windows, macOS, and Linux.

If we use a shebang of #!/usr/bin/env python3, then x.py works for macOS and Linux users, and also works on Windows systems that install Python via the Windows store, but fails to run on Windows systems that install via the official Python installer, requiring users to explicitly invoke Python 3 on the script, and adding friction, support issues, and complexity to our documentation to help users debug that situation.

If we use a shebang of #!/usr/bin/env python, then x.py works for Windows users, fails on some modern macOS systems, works on other modern macOS systems (depending on installation method I think, e.g. homebrew vs Apple), fails on some modern Linux systems, and on macOS and Linux systems where it does work, it might be python2 or python3. So in practice, people often have to explicitly run python3 x.py, which again results in friction, support issues, and complexity in our documentation.

We've even considered things like #!/bin/sh and then writing a shell script hidden inside a Python triple-quoted string, but that doesn't work well on Windows where we can't count on the presence of a shell.

We'd love to write a single shebang that works for all of Windows, macOS, and Linux systems, and doesn't resort in recurring friction or support issues for us across the wide range of systems that our users use.

As far as we can tell, #!/usr/bin/env python3 would work on all platforms, if the Python installer for Windows shipped a python3.exe and handled that shebang by using python3.exe as the interpreter.

Is that something that the official Python installer could consider adding, to make it easy for us to supply cross-platform Python 3 scripts that work out of the box for all our users?

Thank you,
Josh Triplett, on behalf of many Rust team members

[joshtriplett]

Correction to the above evaluation of #!/usr/bin/env python3, based on some retesting on Windows systems:

The failure case we encounter reasonably often involves the official Python installer for Windows, but applies specifically in the case of third-party shells such as MSYS2, which fail with that shebang. #!/usr/bin/env python3 does work with the official Python installer when running from cmd or PowerShell, it just doesn't work from third-party shells.

We have enough users that cases like this come up reasonably often, and it'd be nice to Just Work in those cases too.

Thank you.

[lazka]

(MSYS2 maintainer here) If you run a Python script in MSYS2 bash, the shebang gets interpreted by bash, which looks up in PATH (which by default doesn't include the full Windows PATH to avoid conflicts) and if installed leads to the MSYS2 included CPython to be invoked, either the Cygwin based one, or the MinGW fork, depending on the active MSYS2 environment.

In short, there is no System CPython involved in any of this.

[zooba]

Unfortunately, the only way we can provide a portable shebang is for Linux and macOS users to always use a launcher like on Windows. There's no reliable way for us to modify Windows to do shebangs that look at all native, and the inconsistencies will be worse than the failures.

That said, the new Python install manager provides global py, python and python3 commands that will respect shebangs, and that will soon be the only way for users to get Python, so you can reasonably safely assume the commands will be available everywhere. That's about the best we can offer, at least until the Windows shell¹ adds support.

Footnotes

1. Not strictly in the shell, but that distinction only matters to people familiar with the internals. ↩