lib: adding keep-alive header to download requests

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes

[miladfarca]

Checklist

• commit message follows commit guidelines

Description of change

We have been seeing random failures on AIX in the CI when building node-addon-api:
https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=aix61-ppc64/481/console

Taking a closer look shows the build fails when gyp is trying to download and extract node source tarball. This is done using a stream, data gets decompressed as its being downloaded:

...
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/openssl/opensslconf_no-asm.h
gyp verb extracted file from tarball node-v13.0.0-nightly20190822775048d54c/include/node/zconf.h
Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:183:27) {
errno: -73,
code: 'ECONNRESET',
syscall: 'read'
}
gyp WARN install got an error, rolling back install
...

I came up with a simpler test case to investigate the cause:

consthttps=require('https');vartar=require('tar');constrequest=https.get("https://nodejs.org/download/nightly/v13.0.0-nightly20190822775048d54c/node-v13.0.0-nightly20190822775048d54c-headers.tar.gz",function(response){response.pipe(tar.extract({strip: 1}));});

Running the above on AIX using node test.js causes the same error to appear. It seemed like
the connection was being interrupted while stream was being decompressed.

I noticed requests are being handled by cloudflare. I tried moving the headers.tar.gz file to my personal server with https enabled and then ran the script. It worked with no errors.

I figured cloudflare is most likely dropping the TLS connection once it doesn't hear a tick which happens on AIX as the live tar extraction is interfering with it and is taking longer than other platforms.

I looked at the response headers from cloudflare and the default connection header is set to close. Adding a keep-alive header in my request managed to fix the problem and force a live connection until the stream is done. I also tested this with node-addon-api by modifying gyp and the headers were downloaded successfully on AIX.

I have found a similar ticket from 2017 whit the same issue which might have been caused by cloudflare dropping connections:
#1084

[miladfarca]

/cc @nodejs/platform-ppc

[rvagg]

I don't really get this, it's just performing one request so keep-alive isn't technically necessary, do you think that turning it on is changing the cloudflare timeouts?

[miladfarca]

I'm suspecting there is a timeout which this header is disabling and server continues with sending the stream.

[rvagg]

Well, it's odd, but I guess it can't hurt, so maybe we should give it a go. Any objections @nodejs/node-gyp?

[rvagg]

landed in 0384683, let's cross fingers and see how this goes