build: test Python 3.7 on Travis CI by cclauss · Pull Request #29451 · nodejs/node

cclauss · 2019-09-05T13:59:30Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

sam-github · 2019-09-05T17:36:59Z

Requires #25878 or py3 will fail at configure, so I cherry-picked it in.

Took me a bit to get pyenv working locally on Ubuntu, but while waiting for Travis, I'm in process of reproducing the buildsteps travis should do.

Note, for the pyenv statements to work, anyone else doing this (and me) will need to do export TRAVIS=yes to cause pyenv's python to be chosen over a specific version.

Once I've done this, current state is:

Compile V8

./configure: OK
make -j2 -C out V=1 v8: failing due to Failed to parse config file: Type names and field names cannot be a keyword: 'async', fix in tools: python3 compat for inspector code generator #29340, but after cherry-picking, I get the same issue. I guess the fix is still WIP by @bnoordhuis and @targos , so I'm moving on.

Compile Node.js

make fails with same keyword problem as above
moving on

Test JS Suites

obviously impossible until node.js Compile works

Test C++ Suites

make cctest also has the async keyword problem, so this step is impossible as well

At this point, I think futher progress is blocked until #29340 completes and lands.

cclauss · 2019-09-06T04:33:47Z

As noted in #29340, both the tools and deps version of that file need to be modified.

sam-github · 2019-09-06T04:45:32Z

Thanks, I'll take another shot at this tomorrow.

cclauss · 2019-09-06T10:42:12Z

I added the async change to the deps version of code_generator.py

sam-github · 2019-09-06T19:34:21Z

With the fixes merged in, I can now build and test locally without errors, which is unfortunate, because now I can't troubleshoot locally.

Travis error is:

Building addon in /home/travis/build/nodejs/node/test/addons/01_function_arguments
465Building addon in /home/travis/build/nodejs/node/test/addons/02_callbacks
466/home/travis/build/nodejs/node/tools/build-addons.js:58
467main(process.argv[3]).catch((err) => setImmediate(() => { throw err; }));
468 ^
469
470Error: Command failed: /home/travis/build/nodejs/node/out/Release/node /home/travis/build/nodejs/node/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/travis/build/nodejs/node/test/addons/01_function_arguments
471
472 at ChildProcess.exithandler (child_process.js:295:12)
473 at ChildProcess.emit (events.js:209:13)
474 at maybeClose (internal/child_process.js:1028:16)
475 at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5) {
476 killed: false,
477 code: 1,
478 signal: null,
479 cmd: '/home/travis/build/nodejs/node/out/Release/node /home/travis/build/nodejs/node/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/travis/build/nodejs/node/test/addons/01_function_arguments',
480 stdout: '',
481 stderr: ''
482}
483Makefile:412: recipe for target 'test/addons/.buildstamp' failed

I'm trying to directly run the failing node-gyp command, and interestingly, I see that it uses python2... So, I removed /usr/bin/python2, and see something more interesting:

% PYTHON=/usr/bin/python3 ./out/Release/node deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/travis/build/no
dejs/node/test/addons/01_function_arguments --nodedir=/usr/local/stow/lts/include/
gyp info it worked if it ends with ok gyp info using node-gyp@5.0.3 gyp info using node@13.0.0-pre | linux | x64 gyp WARN chdir /home/travis/build/nodejs/node/test/addons/01_function_arguments is not a directory
gyp ERR! find Python gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python checking Python explicitly set from environment variable PYTHON
gyp ERR! find Python - process.env.PYTHON is "/usr/bin/python3" gyp ERR! find Python - executable path is "/usr/bin/python3"
gyp ERR! find Python - version is "3.7.3"
gyp ERR! find Python - version is 3.7.3 - should be >=2.6.0 <3.0.0 gyp ERR! find Python - THIS VERSION OF PYTHON IS NOT SUPPORTED gyp ERR! find Python checking if "python" can be used gyp ERR! find Python - executable path is "/home/sam/.pyenv/versions/3.7.3/bin/python" gyp ERR! find Python - version is "3.7.3" gyp ERR! find Python - version is 3.7.3 - should be >=2.6.0 <3.0.0
gyp ERR! find Python - THIS VERSION OF PYTHON IS NOT SUPPORTED gyp ERR! find Python checking if "python2" can be used gyp ERR! find Python - "python2" is not in PATH or produced an error

This sparked a theory that Travis doesn't have python2 installed when pyenv global 3.7.1 is used, which is why its failing, and by destroying my local python2, I can now see the same error Travis gets:

% make -j1 V=1 test/addons/.buildstamp
Building addon in /home/sam/w/core/lts/test/addons/01_function_arguments
Building addon in /home/sam/w/core/lts/test/addons/02_callbacks
Building addon in /home/sam/w/core/lts/test/addons/03_object_factory
Building addon in /home/sam/w/core/lts/test/addons/04_function_factory
/home/sam/w/core/lts/tools/build-addons.js:58
main(process.argv[3]).catch((err) => setImmediate(() => { throw err; }));
^
Error: Command failed: /home/sam/w/core/lts/out/Release/node /home/sam/w/core/lts/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/sam/w/core/lts/test/addons/01_function_arguments
at ChildProcess.exithandler (child_process.js:295:12)
at ChildProcess.emit (events.js:209:13)
at maybeClose (internal/child_process.js:1028:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5) {
killed: false,
code: 1,
signal: null,
cmd: '/home/sam/w/core/lts/out/Release/node /home/sam/w/core/lts/deps/npm/node_modules/node-gyp/bin/node-gyp.js rebuild --directory=/home/sam/w/core/lts/test/addons/01_function_arguments',
stdout: '',
stderr: ''
}
make: *** [Makefile:412: test/addons/.buildstamp] Error 1

cclauss · 2019-09-07T05:55:53Z

cp tools/gyp/pylib/gyp/input.py deps/npm/node_modules/node-gyp/gyp/pylib/gyp/
cp tools/gyp/pylib/gyp/generator/make.py deps/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/

cclauss · 2019-09-07T06:54:05Z

https://travis-ci.com/nodejs/node/jobs/232041903#L1008

python tools/test.py -j 2 -p dots --report --mode=release --flaky-tests=dontcare addons js-native-api node-api
tools/test.py:32: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
Total: 115 tests
* 0 tests will be skipped
* 115 tests are expected to pass
* 0 tests are expected to fail that we won't fix
* 0 tests are expected to fail that we should fix
Running 115 tests
...................C..............................
..................................................
...............
=== release test ===
Path: addons/dlopen-ping-pong/test
--- stderr ---
node: ../binding.cc:29: void {anonymous}::LoadLibrary(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `handle != nullptr' failed.
Command: out/Release/node /home/travis/build/nodejs/node/test/addons/dlopen-ping-pong/test.js
--- CRASHED (Signal: 6) ---
===
=== 1 tests failed
=== 1 tests CRASHED
===
The command "python tools/test.py -j 2 -p dots --report --mode=release --flaky-tests=dontcare addons js-native-api node-api" exited with 1.

sam-github · 2019-09-07T17:38:17Z

That's interesting, I'll take a look Monday unless @bnoordhuis gets to it first.

Trott · 2019-09-09T17:59:04Z

I guess the deps changes should be a separate commit from the other ones so that it can be re-cherry-picked onto things if the deps get updated but still don't have these changes?

Trott · 2019-09-09T17:59:52Z

 os: linux
 language: cpp
+# Currently this file can only support one PYTHON_VERSION.
+# To experiment with Python 3, comment out Python 2.7 and uncomment one of the Python 3 versions.


IIUC, with this change, Python 2.7 is already commented out, so this comment needs to be updated?

Trott · 2019-09-09T18:01:40Z

@cclauss Should the work-in-progress label be removed at this point? Or is it still work-in-progress?

sam-github · 2019-09-09T19:29:24Z

Still WIP, it isn't green, and has local copies of changes that are pending in other PRs, or other repos (v8, node-gyp, npm). Its very useful to show what problems will remain even after the other in-progress work completes.

cclauss · 2019-09-09T19:31:20Z

This is still a WIP. It is designed to pass the Travis CI tests on Python 3 but it still fails.

cclauss · 2019-09-09T20:29:47Z

Does anyone know how to resolve the last failing test?

sam-github · 2019-09-09T21:37:34Z

@bnoordhuis any ideas?

bnoordhuis · 2019-09-10T08:39:43Z

I looked into it yesterday but it wasn't immediately obvious to me what went wrong. binding.node tries to dlopen ping.so but somehow it fails.

Time for some printf-style debugging, I suppose. Christian, can you try this?

diff --git a/test/addons/dlopen-ping-pong/binding.cc b/test/addons/dlopen-ping-pong/binding.cc
index 6a6c297b52..02ade757c0 100644
--- a/test/addons/dlopen-ping-pong/binding.cc+++ b/test/addons/dlopen-ping-pong/binding.cc@@ -26,6 +26,7 @@ static ping ping_func;
void LoadLibrary(const FunctionCallbackInfo<Value>& args) {
const String::Utf8Value filename(args.GetIsolate(), args[0]);
void* handle = dlopen(*filename, RTLD_LAZY);
+ if (handle == nullptr) fprintf(stderr, "dlopen(%s): %s\n", *filename, dlerror());
assert(handle != nullptr);
ping_func = reinterpret_cast<ping>(dlsym(handle, "dlopen_ping"));
assert(ping_func != nullptr);

(I tried to push it to your branch but you have collaborator edits disabled, it seems?)

cclauss · 2019-09-10T10:51:01Z

dlopen(/home/travis/build/nodejs/node/test/addons/dlopen-ping-pong/build/Release/ping.so): /home/travis/build/nodejs/node/test/addons/dlopen-ping-pong/build/Release/ping.so: cannot open shared object file: No such file or directory

sam-github · 2019-10-16T01:16:20Z

Rebased onto master now that #29885 has landed.

richardlau

Copying my review comments from #29985 (duplicate?).

sam-github · 2019-10-16T04:00:02Z

I'll clean up the comments and up the python version, but would like to see travis pass. It seems really, really slow... is ccache not shared? Something else? Maybe ccache is reset when .travis.yml changes?

Anyhow, I'll give it overnight to run some more before touching the branch again.

sam-github · 2019-10-16T04:04:36Z

@cclauss I don't think #29985 was needed, this just needed rebasing and pushing. Also, to pass. I can hope 🤞

richardlau · 2019-10-16T04:07:19Z

I'll clean up the comments and up the python version, but would like to see travis pass. It seems really, really slow... is ccache not shared? Something else? Maybe ccache is reset when .travis.yml changes?

https://github.com/nodejs/node/compare/253137b5502a6732eac9f0f899c6577c04338023..2ff431b7a08fab7938a73fb6f10ed76f151035aa#diff-42f385dff7890b3c213d9699bcc350cc suggests v8.h was updated in the force push which is going to force a recompilation of most of V8.

nodejs-github-bot · 2019-10-17T18:23:25Z

CI: https://ci.nodejs.org/job/node-test-pull-request/26085/

sam-github · 2019-10-17T18:23:38Z

@nodejs/python @nodejs/node-gyp PTAL

sam-github · 2019-10-17T20:02:17Z

smartos out of memory:

tools/doc/generate.js --node-version=v13.0.0 --apilinks=out/doc/apilinks.json doc/api/url.md --output-directory=out/doc/api\""; exit 1; fi;
11:33:46 11:33:46 11:33:46 #
11:33:46 # Fatal error in , line 0
11:33:46 # Fatal process out of memory: Zone
11:33:46 #
11:33:46 #
11:33:46 #
11:33:46 #FailureMessage Object: fffffbffede7a500

Windows is an instance of #30011

Resuming the build.

nodejs-github-bot · 2019-10-17T20:02:50Z

CI: https://ci.nodejs.org/job/node-test-pull-request/26087/

sam-github · 2019-10-17T20:04:21Z

@nodejs/platform-smartos FYI - #29451 (comment) I don't know how frequent that is, and hope it will be ephemral, but just to let you know.

Request Python 3 with pyenv and ensure that python3 is used by Makefile to run Python scripts. PR-URL: #29451 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org>

sam-github · 2019-10-18T18:41:40Z

Landed in d594a9a 🎉

Request Python 3 with pyenv and ensure that python3 is used by Makefile to run Python scripts. PR-URL: #29451 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org>

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Sep 5, 2019

cclauss mentioned this pull request Sep 5, 2019
tools/test.py is not yet Python 3 compatible #29246
Closed

cclauss added the python PRs and issues that require attention from people who are familiar with Python. label Sep 5, 2019

sam-github added the wip Issues and PRs that are still a work in progress. label Sep 5, 2019

sam-github force-pushed the Travis-test-Py36-or-Py37 branch from cfe37bd to 0d60361Compare September 5, 2019 16:57

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 567ca38 to b5fd477Compare September 6, 2019 10:37

cclauss mentioned this pull request Sep 6, 2019
build: add comment to .travis.yml on how to test Py3 #29473
Closed
4 tasks

cclauss changed the title ~~build: explain how to test Py3.6 & Py3.7 on Travis~~build: test Python 3.7 on Travis CISep 6, 2019

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 7d70614 to 014f468Compare September 7, 2019 05:55

Trott reviewed Sep 9, 2019
View reviewed changes

sam-github force-pushed the Travis-test-Py36-or-Py37 branch from 2ba9e62 to b535e9cCompare September 9, 2019 21:34

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from b535e9c to 23ce029Compare September 9, 2019 21:35

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 23ce029 to 5a6e21dCompare September 9, 2019 21:48

cclauss force-pushed the Travis-test-Py36-or-Py37 branch from 5a6e21d to 9ab7107Compare September 10, 2019 10:28

sam-github force-pushed the Travis-test-Py36-or-Py37 branch from 253137b to 2ff431bCompare October 16, 2019 01:13

rvagg reviewed Oct 16, 2019
View reviewed changes

Comment thread.travis.yml

richardlau reviewed Oct 16, 2019
View reviewed changes

Comment thread.travis.yml Outdated
Comment thread.travis.yml Outdated

fixup: remove EXPERIMENTAL_NODE_GYP_PYTHON3="yes"
77fb266

sam-github removed the wip Issues and PRs that are still a work in progress. label Oct 17, 2019

richardlau approved these changes Oct 17, 2019
View reviewed changes

sam-github approved these changes Oct 17, 2019
View reviewed changes

sam-github mentioned this pull request Oct 17, 2019
flaky test: test-http-dump-req-when-res-ends on windows #30011
Closed

rvagg approved these changes Oct 17, 2019
View reviewed changes

sam-github closed this Oct 18, 2019

MylesBorins mentioned this pull request Oct 23, 2019
v13.0.1 proposal #30081
Merged

targos mentioned this pull request Nov 10, 2019
v12.13.1 release proposal #30352
Merged

targos added the dont-land-on-v12.x label Nov 11, 2019

devsnek deleted the Travis-test-Py36-or-Py37 branch May 21, 2020 17:27

Uh oh!

Uh oh!

Conversation

cclauss commented Sep 5, 2019• edited by sam-github Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

sam-github commented Sep 5, 2019

Compile V8

Compile Node.js

Test JS Suites

Test C++ Suites

Uh oh!

cclauss commented Sep 6, 2019

Uh oh!

sam-github commented Sep 6, 2019

Uh oh!

cclauss commented Sep 6, 2019

Uh oh!

sam-github commented Sep 6, 2019

Uh oh!

cclauss commented Sep 7, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cclauss commented Sep 7, 2019

Uh oh!

sam-github commented Sep 7, 2019

Uh oh!

Trott commented Sep 9, 2019

Uh oh!

TrottSep 9, 2019

Choose a reason for hiding this comment

Uh oh!

Trott commented Sep 9, 2019

Uh oh!

sam-github commented Sep 9, 2019

Uh oh!

cclauss commented Sep 9, 2019

Uh oh!

cclauss commented Sep 9, 2019

Uh oh!

sam-github commented Sep 9, 2019

Uh oh!

bnoordhuis commented Sep 10, 2019

Uh oh!

cclauss commented Sep 10, 2019

Uh oh!

sam-github commented Oct 16, 2019

Uh oh!

Uh oh!

richardlau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sam-github commented Oct 16, 2019

Uh oh!

sam-github commented Oct 16, 2019

Uh oh!

richardlau commented Oct 16, 2019

Uh oh!

nodejs-github-bot commented Oct 17, 2019

Uh oh!

sam-github commented Oct 17, 2019

Uh oh!

sam-github commented Oct 17, 2019

Uh oh!

nodejs-github-bot commented Oct 17, 2019

Uh oh!

sam-github commented Oct 17, 2019

Uh oh!

sam-github commented Oct 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

cclauss commented Sep 5, 2019•
edited by sam-github
Loading

cclauss commented Sep 7, 2019•
edited
Loading