Consider using the "py.exe" launcher to find Python 3 as well, not just for Python 2

[DeeDeeG]

• Node Version: Applies to any versions of Node/NPM, but tested on Node 10.20.1 with npm 6.14.4
• Platform: This is about Windows, in general. Tested on Windows 10.
• Compiler: N/A; this is just about finding a Python binary. No compiler needed on the system to find a Python binary.
• Module: Nothing, just did findPython = require(./lib/find-python) in a little standalone .js script to test Python detection.

My script ("fpy.js") (to be saved to/called from the node-gyp repo's main folder, like so: "node fpy.js".)

findPython = require('./lib/find-python')

findPython(null, function (err, found) {
 if (err) {
 console.error(err)
 } else {
 python = found
 console.log('python was found: ' + python)
 }
})

Verbose output (from npm or node-gyp):

node.exe ./fpy.js
ERR! find Python
ERR! find Python Python is not set from command line or npm configuration
ERR! find Python Python is not set from environment variable PYTHON
ERR! find Python checking if "python3" can be used
ERR! find Python - "python3" is not in PATH or produced an error
ERR! find Python checking if "python" can be used
ERR! find Python - "python" is not in PATH or produced an error
ERR! find Python checking if "python2" can be used
ERR! find Python - "python2" is not in PATH or produced an error
ERR! find Python checking if Python is C:\Python37\python.exe
ERR! find Python - "C:\Python37\python.exe" could not be run
ERR! find Python checking if Python is C:\Python27\python.exe
ERR! find Python - "C:\Python27\python.exe" could not be run
ERR! find Python checking if the py launcher can be used to find Python 2
ERR! find Python - "py.exe" is not in PATH or produced an error
ERR! find Python
ERR! find Python **********************************************************
ERR! find Python You need to install the latest version of Python.
ERR! find Python Node-gyp should be able to find and use Python. If not,
ERR! find Python you can try one of the following options:
ERR! find Python - Use the switch --python="C:\Path\To\python.exe"
ERR! find Python (accepted by both node-gyp and npm)
ERR! find Python - Set the environment variable PYTHON
ERR! find Python - Set the npm configuration variable python:
ERR! find Python npm config set python "C:\Path\To\python.exe"
ERR! find Python For more information consult the documentation at:
ERR! find Python https://github.com/nodejs/node-gyp#installation
ERR! find Python **********************************************************
ERR! find Python
Error: Could not find any Python installation to use
 at PythonFinder.fail (C:\Users\[User]\node-gyp\lib\find-python.js:307:47)
 at PythonFinder.runChecks (C:\Users\[User]\node-gyp\lib\find-python.js:136:21)
 at PythonFinder.<anonymous> (C:\Users\[User]\node-gyp\lib\find-python.js:205:18)
 at PythonFinder.execFileCallback (C:\Users\[User]\node-gyp\lib\find-python.js:271:16)
 at ChildProcess.exithandler (child_process.js:301:5)
 at ChildProcess.emit (events.js:198:13)
 at maybeClose (internal/child_process.js:982:16)
 at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)

---

Intro, Background, and the Gist of it

Hi folks, I'm wondering if the node-gyp team would be interested in using the py.exe launcher to find not only/exclusively Python 2, like it does now, but potentially Python 3 as well.

(I just installed Python 3.8 from python.org, in the default install location, and node-gyp can't find it. py.exe can find it, though.)

Apparently, judging by the commit history, using the Python launcher strictly to find Python 2 dates back just over four years ago, to BEFORE node-gyp had any support for Python 3. From here: #894 (The latest published version of node-gyp would have been v3.3.1 at the time, judging by the release history: https://github.com/nodejs/node-gyp/releases?after=v3.5.0)

Thus, I assume the justification for only finding Python 2 at that time would have been the following: "node-gyp can't work with Python 3, so finding Python 3 would be counter-productive! We must only find Python 2..." (That's no longer true, and I think it might be nice to find Python 3 with such a convenient helper/launcher like py.exe.)

---

About the py.exe Launcher, and the installation location of Python on Windows

The py.exe launcher is quite useful for finding Python. It claims to be able to find the version downloadable from python.org AND the Python downloadable from the Windows Store. Source: https://docs.python.org/3/using/windows.html#launcher) It comes with Python installers for Windows since Python 3.3.

The Python situation on Windows is a little odd now, because it seems it used to be saved under something like C:\Python[Maj[Min]\python.exe (for example: C:\Python37), but recent installers save to somewhere more like this: 'C:\Users\[User]\AppData\Local\Programs\Python\Python38-32' and in fact, the location can be customized during the install wizard. So it could end up anywhere,
I guess.

I think the launcher might be the most reliable way to find Python on Windows these days. And in any case, it makes an excellent fallback. I only find that it is a shame that it isn't being used to find Python 3.

Indeed, going to Python.org, downloading Python 3.8.x and installing it will result in a Python installation that node-gyp can't find. I would like to approximate node-gyp's Python finding algorithm (or require its find-python.js file directly), but not being able to find the default install location from python.org seems somehow not right. I didn't want to re-implement what seems like a (small) oversight in the repo I'm working on.

I do think the Python from the Windows Store might end up on the path as python and/or python3, but I haven't tested or confirmed this.

---

Suggested Solution / Notes on an Implementation

Sorry I'm not very good with JS, or I'd whip up an implementation for a PR. But I think you could really just drop the -2 here, and update the comment above that, and that should do it.

(If that solution would be sufficient, or you'd like to see it run it through the CI tests and such, I would be GLAD to whip that up as a PR.)

From a quick read of the documentation page, the py.exe launcher seems to prefer Python 3 over Python 2, so just dropping the -2 flag might not be appropriate for for the node-gyp 5.x branch (which, from what I can see, generally prefers Python 2 over Python 3.)

[cclauss]

Our automated tests pass on Python 2.7, 3.6, 3.7, and 3.8 and Python 3 Is supported on Windows.

Why find Python 2 at all? It died on 1/1/2020. Let’s just find supported versions of Python and ignore EOL versions of Python.

[DeeDeeG]

Hmm. Fair enough.

I was thinking of posting separate fixes for node-gyp's 6.x branch and 5.x branch, and Python2 would arguably still be relevant to the 5.x branch, since that is the Python2-themed conservative version of 6.x (and used in npm for now).

(Edit: And to tell the whole story, there was a final release of Python 2.7.x in April 2020: https://www.python.org/psf/press-release/pr20191220/ https://www.python.org/doc/sunset-python-2/ but on closer reading their announcement/official position, they seem pretty clear they still want users to move away from Python 2.)

[cclauss]

The last three or four versions of node-gyp’s 5.x branch support Python 3.

[DeeDeeG]

I only mean, node-gyp v5.1.0 is basically node-gyp v6.1.0 but the order it looks for Python is:

• python, python2, python3

Whereas in node-gyp v6 it's

• python3, python, python2

If this were the among the last changes to the 5.x branch, it would make that branch more python3-aware and encourage users of today's npm to be able to use Python 3. [Edit: Assuming npm takes the patch-level bump of node-gyp v5.1.1 or v5.1.2 or whatever it would end up being.]

[DeeDeeG]

FWIW the py.exe launcher deliberately supports finding Python 2.x and Python 3.x. There is no way to stop it supporting Python 2 short of patching it over at the Python project.

I just mean node-gyp should stop running it with the explicit -2 flag.

(On node-gyp 5.x, I propose that it can be run first with the -2 flag and then without. Because the whole find-python logic on that branch is oriented toward preferring Python 2 over Python 3 if both are on the system. And I suppose that's because preferring Python 3 was designated a breaking/semver-major change for the 6x branch. Reading the CHANGELOG, I see Python 3 as essentially the big semver-major change in node-gyp 6.x.)

[cclauss]

There are several documented ways to get node-gyp to use the Python that you want.
Perhaps it would be better to encourage npm/cli to upgrade to node-gyp v6 so we can retire v5.

[cclauss]

I think overall that your direction is the right one. Going forward node-gyp should use py.exe to find Python on Windows. Our custom code for this is no longer useful.

[DeeDeeG]

This is a basic implementation for the 6.x branch: DeeDeeG@bb768d0

If there's something cleverer than that, it would unfortunately be a bit past my skill level in JS. I'm very much a beginner with JS.

[bzoz]

Some people are using python from the depot-tools, will the py.exe launcher work for them?

[cclauss]

Is depot-tools compatible with current versions of Python?

[DeeDeeG]

Some people are using python from the depot-tools, will the py.exe launcher work for them?

Short answer: No. It appears that properly setting up depot_tools puts both python.bat and python3.bat on the PATH. These are batch scripts to locate the actual python or python3 binaries within depot_tools. Apparently py.exe doesn't notice these or know what to do with these, and neither does node-gyp's find-python.js at the moment.

---

I just tried installing depot_tools to see for myself. Using the "latest instructions" from here ("latest" according to this page).

• According to the instructions, the depot_tools folder becomes the first item in the user's PATH
• depot_tools\python.bat is a little finder utility that ends up on the PATH
  • In my experience, this is able to find depot_tools\bootstrap-3_8_0_chromium_8_bin\python\bin\python.exe
    • This is a Python 2.7.17 binary, at the time I'm writing this.
• depot_tools\python3.bat is another finder script, but for Python 3.
  • This finds depot_tools\bootstrap-3_8_0_chromium_8_bin\python3\bin\python3.exe
    • Python 3.8.0, at the time I'm writing this.

Doing where python in CMD.exe finds the depot_tools\python.bat launcher (doing just python runs the .bat file and gets redirected to execute --> depot_tools\bootstrap-3_8_0_chromium_8_bin\python\bin\python.exe). (Likewise, doing where python3 finds the depot_tools\python3.bat launcher, and doing just python3 runs the .bat launcher which redirects to execute --> depot_tools\bootstrap-3_8_0_chromium_8_bin\python3\bin\python3.exe.)

Neither the py.exe launcher nor the rest of this repo's find-python.js are able to identify or use these launcher/helper .bat files properly.

Apparently having the depot_tools set up and on your PATH is not enough to have py.exe or node-gyp find these Python versions.

Weirdly enough, doing SET PYTHON=path\to\depot_tools\python.bat is enough for node-gyp to find depot_tools\python.bat and not get put off by the redirect happening to depot_tools\bootstrap-3_8_0_chromium_8_bin\python\bin\python.exe. So, setting the PYTHON environment variable remains a viable workaround, as always.

I wonder if it would be worth looking into this and correcting it so node-gyp's find-python.js can use these python[3].bat launchers. Given that they are already on the PATH, it might be a small change in the node-gyp code. I'm not sure how many people have Python from depot_tools, but if it's a large number of users this could be a nice thing. As mentioned, they already can work if specified in the PYTHON env variable, so I don't see technically why this isn't happening automatically.

---

I note that the py.exe launcher seems to ignore the PATH entirely?? Because when I put the valid python.exe and python3.exe from depot_tools directly on my PATH, (e.g. SET PATH=%PATH%;depot_tools\bootstrap-3_8_0_chromium_8_bin\python\bin) py -0 (which lists Python binaries the launcher can find) doesn't list them. But if I do where python or where python3, the ones I manually added from depot_tools are there.

[DeeDeeG]

Here's a proof of concept implementation against the v5.x branch: DeeDeeG@b2119e3

I have tested it, and it works as intended, but it also does appear to break the tests.