Random HTTP 500 statuses or ECONNRESET when node-gyp fetches dependencies

[rchennafi]

This is occurring across multiple node versions, architectures and system types. Our build system regularly runs node gyp in its build for our NPM packages across every version of nodejs and about 5% of builds are subject to a 500 status failure or an ECONNRESET when setting up node-gyp.

This strikes me as an issue with the server which hosts the node gyp files. Is there possibly a way around downloading these node-gyp dependencies each time? Or maybe something that can be done to fix the instability there?

These builds all took place inside of Amazon server instances.

Here is an instance on node 14 with node gyp 7.1.2 on a Windows Server 2019 machine.

04:08:09 gyp info ok 
04:08:13 Now using node v14.21.2 (64-bit)
04:08:19 gyp info it worked if it ends with ok
04:08:19 gyp info using node-gyp@7.1.2
04:08:19 gyp info using node@14.21.2 | win32 | x64
04:08:19 gyp info find Python using Python version 3.8.5 found at "C:\Python38\python3.exe"
04:08:23 gyp http GET https://nodejs.org/download/release/v14.21.2/node-v14.21.2-headers.tar.gz
04:08:23 gyp http 200 https://nodejs.org/download/release/v14.21.2/node-v14.21.2-headers.tar.gz
04:08:25 gyp http GET https://nodejs.org/download/release/v14.21.2/SHASUMS256.txt
04:08:25 gyp http GET https://nodejs.org/download/release/v14.21.2/win-x86/node.lib
04:08:25 gyp http GET https://nodejs.org/download/release/v14.21.2/win-x64/node.lib
04:08:25 gyp http GET https://nodejs.org/download/release/v14.21.2/win-arm64/node.lib
04:08:25 gyp http 200 https://nodejs.org/download/release/v14.21.2/SHASUMS256.txt
04:08:26 gyp http 500 https://nodejs.org/download/release/v14.21.2/win-arm64/node.lib
04:08:26 gyp WARN install got an error, rolling back install
04:08:26 gyp ERR! configure error 
04:08:26 gyp ERR! stack Error: 500 status code downloading arm64 node.lib
04:08:26 gyp ERR! stack at Request.<anonymous> (C:\nvm\v14.21.2\node_modules\node-gyp\lib\install.js:335:22)
04:08:26 gyp ERR! stack at Request.emit (events.js:412:35)
04:08:26 gyp ERR! stack at Request.onRequestResponse (C:\nvm\v14.21.2\node_modules\node-gyp\node_modules\request\request.js:1059:10)
04:08:26 gyp ERR! stack at ClientRequest.emit (events.js:400:28)
04:08:26 gyp ERR! stack at HTTPParser.parserOnIncomingClient (_http_client.js:647:27)
04:08:26 gyp ERR! stack at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)
04:08:26 gyp ERR! stack at TLSSocket.socketOnData (_http_client.js:515:22)
04:08:26 gyp ERR! stack at TLSSocket.emit (events.js:400:28)
04:08:26 gyp ERR! stack at addChunk (internal/streams/readable.js:293:12)
04:08:26 gyp ERR! stack at readableAddChunk (internal/streams/readable.js:267:9)
04:08:26 gyp ERR! System Windows_NT 10.0.17763
04:08:26 gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--msvs_version=2017" "--devdir=C:\\tmp\\.gyp"

Here is an instance of it occurring on Linux ARM64 with electron & node 16 and node-gyp 9.2.0.

02:06:10 gyp info it worked if it ends with ok
02:06:10 gyp info using node-gyp@9.2.0
02:06:10 gyp info using node@16.19.0 | linux | arm64
02:06:10 gyp info find Python using Python version 3.6.8 found at "/usr/bin/python3"
02:06:10 gyp http GET https://electronjs.org/headers/v9.0.0/node-v9.0.0-headers.tar.gz
02:06:11 gyp WARN install got an error, rolling back install
02:06:11 gyp ERR! configure error 
02:06:11 gyp ERR! stack FetchError: request to https://www.electronjs.org/headers/v9.0.0/node-v9.0.0-headers.tar.gz failed, reason: read ECONNRESET
02:06:11 gyp ERR! stack at ClientRequest.<anonymous> (/home/pdftron/.nvm/versions/node/v16.19.0/lib/node_modules/node-gyp/node_modules/minipass-fetch/lib/index.js:130:14)
02:06:11 gyp ERR! stack at ClientRequest.emit (node:events:513:28)
02:06:11 gyp ERR! stack at TLSSocket.socketErrorListener (node:_http_client:494:9)
02:06:11 gyp ERR! stack at TLSSocket.emit (node:events:525:35)
02:06:11 gyp ERR! stack at emitErrorNT (node:internal/streams/destroy:157:8)
02:06:11 gyp ERR! stack at emitErrorCloseNT (node:internal/streams/destroy:122:3)
02:06:11 gyp ERR! stack at processTicksAndRejections (node:internal/process/task_queues:83:21)
02:06:11 gyp ERR! System Linux 5.10.0-12-cloud-arm64
02:06:11 gyp ERR! command "/home/pdftron/.nvm/versions/node/v16.19.0/bin/node" "/home/pdftron/.nvm/versions/node/v16.19.0/bin/node-gyp" "rebuild" "--target=v9.0.0" "--dist-url=https://electronjs.org/headers" "--openssl_fips=" "--arch=arm"
02:06:11 gyp ERR! cwd /home/admin/jenkins/workspace/Node_Linux_Electron_Arm64_master/WebPDFWorker/Node.js/Packaging
02:06:11 gyp ERR! node -v v16.19.0
02:06:11 gyp ERR! node-gyp -v v9.2.0
02:06:11 gyp ERR! not ok 

And on Linux x64 with node 9, node-gyp 5.1.1

01:50:03 Now using node v9.11.2 (npm v5.6.0)
01:50:03 gyp info it worked if it ends with ok
01:50:03 gyp info using node-gyp@5.1.1
01:50:03 gyp info using node@9.11.2 | linux | x64
01:50:03 gyp info find Python using Python version 2.7.5 found at "/usr/bin/python"
01:50:03 gyp http GET https://nodejs.org/download/release/v9.11.2/node-v9.11.2-headers.tar.gz
01:50:03 gyp http 500 https://nodejs.org/download/release/v9.11.2/node-v9.11.2-headers.tar.gz
01:50:03 gyp WARN install got an error, rolling back install
01:50:03 gyp ERR! configure error 
01:50:03 gyp ERR! stack Error: 500 response downloading https://nodejs.org/download/release/v9.11.2/node-v9.11.2-headers.tar.gz
01:50:03 gyp ERR! stack at Request.<anonymous> (/home/pdftron/.nvm/versions/node/v9.11.2/lib/node_modules/node-gyp/lib/install.js:188:14)
01:50:03 gyp ERR! stack at Request.emit (events.js:185:15)
01:50:03 gyp ERR! stack at Request.onRequestResponse (/home/pdftron/.nvm/versions/node/v9.11.2/lib/node_modules/node-gyp/node_modules/request/request.js:1059:10)
01:50:03 gyp ERR! stack at ClientRequest.emit (events.js:180:13)
01:50:03 gyp ERR! stack at HTTPParser.parserOnIncomingClient (_http_client.js:540:21)
01:50:03 gyp ERR! stack at HTTPParser.parserOnHeadersComplete (_http_common.js:117:17)
01:50:03 gyp ERR! stack at TLSSocket.socketOnData (_http_client.js:445:20)
01:50:03 gyp ERR! stack at TLSSocket.emit (events.js:180:13)
01:50:03 gyp ERR! stack at addChunk (_stream_readable.js:274:12)
01:50:03 gyp ERR! stack at readableAddChunk (_stream_readable.js:261:11)
01:50:03 gyp ERR! System Linux 5.10.0-19-cloud-amd64
01:50:03 gyp ERR! command "/home/pdftron/.nvm/versions/node/v9.11.2/bin/node" "/home/pdftron/.nvm/versions/node/v9.11.2/bin/node-gyp" "rebuild"
01:50:03 gyp ERR! cwd /home/admin/jenkins/workspace/PDFNetNode_Linux_master/WebPDFWorker/Node.js/Packaging
01:50:03 gyp ERR! node -v v9.11.2
01:50:03 gyp ERR! node-gyp -v v5.1.1
01:50:03 gyp ERR! not ok

These were just the instances I was able to find within the last 2 weeks of our daily build system. If you want more specific details about each machine I can provide those. The Linux builds are all run within Docker containers.

[cclauss]

Python 3.6 (and soon 3.7) is no longer supported... https://devguide.python.org/versions

Node versions < 16 are no longer supported... https://github.com/nodejs/Release#nodejs-release-working-group

[rchennafi]

@cclauss
This occurs across all version and types regardless of node/python versions sporadically. Here is a log for Node 19 & Python 3.8.5 on Windows.

03:35:25 gyp info ok 
03:35:29 Now using node v19.5.0 (64-bit)
03:35:41 gyp info it worked if it ends with ok
03:35:41 gyp info using node-gyp@9.3.1
03:35:41 gyp info using node@19.5.0 | win32 | x64
03:35:44 gyp info find Python using Python version 3.8.5 found at "C:\Python38\python3.exe"
03:35:52 gyp http GET https://nodejs.org/download/release/v19.5.0/node-v19.5.0-headers.tar.gz
03:35:53 gyp http 200 https://nodejs.org/download/release/v19.5.0/node-v19.5.0-headers.tar.gz
03:37:14 gyp http GET https://nodejs.org/download/release/v19.5.0/SHASUMS256.txt
03:37:14 gyp http GET https://nodejs.org/download/release/v19.5.0/win-x86/node.lib
03:37:14 gyp http GET https://nodejs.org/download/release/v19.5.0/win-x64/node.lib
03:37:14 gyp http GET https://nodejs.org/download/release/v19.5.0/win-arm64/node.lib
03:37:14 gyp http 404 https://nodejs.org/download/release/v19.5.0/win-arm64/node.lib
03:37:14 gyp http 200 https://nodejs.org/download/release/v19.5.0/win-x64/node.lib
03:37:14 gyp http 200 https://nodejs.org/download/release/v19.5.0/SHASUMS256.txt
03:37:15 gyp http 200 https://nodejs.org/download/release/v19.5.0/win-x86/node.lib
03:42:22 gyp WARN install got an error, rolling back install
03:42:22 gyp ERR! configure error 
03:42:22 gyp ERR! stack Error: aborted
03:42:22 gyp ERR! stack at connResetException (node:internal/errors:717:14)
03:42:22 gyp ERR! stack at TLSSocket.socketCloseListener (node:_http_client:456:19)
03:42:22 gyp ERR! stack at TLSSocket.emit (node:events:524:35)
03:42:22 gyp ERR! stack at node:net:316:12
03:42:22 gyp ERR! stack at TCP.done (node:_tls_wrap:588:7)
03:42:22 gyp ERR! System Windows_NT 10.0.17763
03:42:22 gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--msvs_version=2017" "--devdir=C:\\tmp\\.gyp"
03:42:22 gyp ERR! cwd C:\Jenkins\workspace\PDFNetNode_Windows_master\WebPDFWorker\Node.js\Packaging
03:42:22 gyp ERR! node -v v19.5.0
03:42:22 gyp ERR! node-gyp -v v9.3.1
03:42:22 gyp ERR! not ok 

[cclauss]

03:37:14 gyp http 404 https://nodejs.org/download/release/v19.5.0/win-arm64/node.lib

@StefanStojanovic Any ideas?

[StefanStojanovic]

03:37:14 gyp http 404 https://nodejs.org/download/release/v19.5.0/win-arm64/node.lib

This is expected because win-arm64 is part of official releases starting with v19.9.0. However, if HTTP status code 403 or 404 is received for downloading node.lib that should not be a problem according to this code.

03:37:15 gyp http 200 https://nodejs.org/download/release/v19.5.0/win-x86/node.lib
03:42:22 gyp WARN install got an error, rolling back install

Based on this part of the log, an installer error happened 5 minutes after downloading all of the node.lib files and SHASUMS256.txt

[rchennafi]

Another one here for windows and supported versions. This occurred today.

06:05:52 Now using node v18.14.0 (64-bit)
06:06:04 gyp info it worked if it ends with ok
06:06:04 gyp info using node-gyp@9.3.1
06:06:04 gyp info using node@18.14.0 | win32 | x64
06:06:05 gyp info find Python using Python version 3.8.5 found at "C:\Python38\python3.exe"
06:06:10 gyp http GET https://nodejs.org/download/release/v18.14.0/node-v18.14.0-headers.tar.gz
06:06:11 gyp http 200 https://nodejs.org/download/release/v18.14.0/node-v18.14.0-headers.tar.gz
06:06:20 gyp http GET https://nodejs.org/download/release/v18.14.0/SHASUMS256.txt
06:06:20 gyp http GET https://nodejs.org/download/release/v18.14.0/win-x86/node.lib
06:06:20 gyp http GET https://nodejs.org/download/release/v18.14.0/win-arm64/node.lib
06:06:20 gyp http GET https://nodejs.org/download/release/v18.14.0/win-x64/node.lib
06:06:20 gyp http 200 https://nodejs.org/download/release/v18.14.0/SHASUMS256.txt
06:06:20 gyp http 500 https://nodejs.org/download/release/v18.14.0/win-arm64/node.lib
06:06:20 gyp WARN install got an error, rolling back install
06:06:21 gyp ERR! configure error 
06:06:21 gyp ERR! stack Error: 500 status code downloading arm64 node.lib
06:06:21 gyp ERR! stack at C:\nvm\v18.14.0\node_modules\node-gyp\lib\install.js:256:17
06:06:21 gyp ERR! stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
06:06:21 gyp ERR! stack at async Promise.all (index 2)
06:06:21 gyp ERR! stack at async go (C:\nvm\v18.14.0\node_modules\node-gyp\lib\install.js:189:5)
06:06:21 gyp ERR! stack at async install (C:\nvm\v18.14.0\node_modules\node-gyp\lib\install.js:62:18)
06:06:21 gyp ERR! System Windows_NT 10.0.17763
06:06:21 gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--msvs_version=2017" "--devdir=C:\\tmp\\.gyp"
06:06:21 gyp ERR! cwd C:\Jenkins\workspace\PDFNetNode_Windows_9.5\WebPDFWorker\Node.js\Packaging
06:06:21 gyp ERR! node -v v18.14.0
06:06:21 gyp ERR! node-gyp -v v9.3.1
06:06:21 gyp ERR! not ok 

[cclauss]

Is this now only a Window issue?

[rchennafi]

It is most consistent on Windows and Windows is probably the main pain point. The last few logs I've linked are all x64 so not specifically ARM either. Last night we had 4 windows builds all fail around 4:00 AM PST across different node versions with various ECONNRESET and 500 errors.

Some context here, we have node builds for electron and node each for Mac, Windows and Linux. This ends up being about 18 builds per night each running node 8->19 and for electron, 11->22.

[StefanStojanovic]

Status code 500, unlike 403 and 404 fails the installation, as seen from #2847 (comment) since no time passes between those two logs. The error is thrown from here.

I'm not sure this is a Windows-specific issue, especially not an ARM-on-Windows issue. Getting HTTP 500 and ECONNRESET seems more like an issue in communication with the server from which the binaries are downloaded than anything else.

[StefanStojanovic]

A quick update on this topic: I've investigated this issue more and found a way to improve Windows installation to decrease network traffic. I have a PoC, but since it changes the install.js script, I'm waiting for #2846 to land. Afterward, I'll rebase, resolve potential conflicts and open the PR.