What / Why
When invoking npm ci, the command fails with premature close and little else.
When
- invoking
npm ci and a git dependency with a prepare script is included.
Where
- Our internal Jenkins servers.
- Internal packages are consumed over git. Public packages are fine.
How
Current Behavior
When invoking npm ci, our org sees this error:
[2020-04-15T18:53:51.235Z] + npm ci
[2020-04-15T18:53:56.677Z] npm ERR! premature close
[2020-04-15T18:54:04.979Z] [2020-04-15T18:54:04.979Z] npm ERR! A complete log of this run can be found in:
[2020-04-15T18:54:04.979Z] npm ERR! /home/jenkins/.npm/_logs/2020-04-15T18_53_56_371Z-debug.log
Steps to Reproduce
This is sticky, but here's my best shot:
- Have a repository that consumes another via
git. - The consumed repository has a
prepare script. Doesn't matter what's in it. - Have a sufficient number of CA certificates? I'm less confident here.
- Run
npm ci. - Observe. Weep.
Expected Behavior
npm ci should succeed, given the prepare script and all of other packages are healthy.
Who
My workplace, but I am happy to be the primary contact.
References
- Perhaps coincidentally related to fd0a802
Other notes
I spent some time on this and it's possible more than one bug will fall out of it. The logging here doesn't seem useful (a bug of its own), but fixing the logging won't get down to the root cause. I intend to track the root cause in this ticket, and may file another for the logging issue.
When we dial up the logging to silly, we more or less see the same thing. We did prune our dependencies until we found the foremost ones involved. They are internal git based dependencies. I can furnish the verbose logs if desired, but I will omit them from the initial report since I believe they will only add noise. The last log seen from the problematic packages is prepareGitDep.
prepareGitDep <repo-name>@git+ssh://git@<repo-url>.git#<commit-hash>: installing devDeps and running prepare script.
There are a lot of other logs between this and other things - I assume this is due to the asynchronous manner of package installation.
The git dependency in question has a prepare script declared in its package.json. If I remove prepare, everything works. If I replace the prepare script with something rudimentary such as "echo 'hi'", "true", or even "" I still get the error above.
This may be noise: We noticed this in 6.14.4 from 6.13.4 - when our Node version transitioned from 12.16.2 to 12.16.3. I was able to walk commits and found the problem reliably appeared at this commit. All it does is points cacache to the _cacache subdirectory, whereas beforehand there was no subdirectory. Later we were able to reproduce the error in 6.13.4, so I think the _cacache change might exacerbate the issue, but is not the direct cause.
I made some direct edits to my local copy of npm and found that if, in pack.js, I change cliArgs to just be [] that everything Just Works(tm). I don't believe this is a sustainable fix though. Further debugged revealed the issue to be E2BIG - an error that a single command line argument exceeded the kernel's restriction. Ours is set to 2097152, which I understand to be the Linux default - and is restricted by the kernel's page size. You can view your own with getconf ARG_MAX at your terminal. I inspected the cliArgs and found that ca was set to the cert data of every certificate on the machine. I perused these and found a standard assortment of CA certs, intermediate certificates, etc. Our organization's internal certificates were included. I assume but have not yet verified that this is simply reading from the local machine's OpenSSL's certificate list - I know that we don't set it explicitly. .npmrc has only cafile=/etc/pki/tls/certs/ca-bundle.crt in relation to certificates. The build machine where this behavior is observed is running CentOS 7.7.1908. Most of us use Macs at the workplace, and we do not see this error on them with the same node/npm versions.
Since this restriction applies to a single argument, I chopped up ca into multiple ca[] arguments. This seems to work for pack.js but it kicks the can down the road. I suspect that other parts of npm need a similar treatment, but the processes have either become too opaque for me to trace much further, or I my simian mind is too simple and unfamiliar with npm innards.
What / Why
When invoking
npm ci, the command fails withpremature closeand little else.When
npm ciand a git dependency with a prepare script is included.Where
How
Current Behavior
When invoking
npm ci, our org sees this error:Steps to Reproduce
This is sticky, but here's my best shot:
git.preparescript. Doesn't matter what's in it.npm ci.Expected Behavior
npm cishould succeed, given thepreparescript and all of other packages are healthy.Who
My workplace, but I am happy to be the primary contact.
References
Other notes
I spent some time on this and it's possible more than one bug will fall out of it. The logging here doesn't seem useful (a bug of its own), but fixing the logging won't get down to the root cause. I intend to track the root cause in this ticket, and may file another for the logging issue.
When we dial up the logging to
silly, we more or less see the same thing. We did prune our dependencies until we found the foremost ones involved. They are internal git based dependencies. I can furnish the verbose logs if desired, but I will omit them from the initial report since I believe they will only add noise. The last log seen from the problematic packages isprepareGitDep.There are a lot of other logs between this and other things - I assume this is due to the asynchronous manner of package installation.
The git dependency in question has a
preparescript declared in itspackage.json. If I removeprepare, everything works. If I replace thepreparescript with something rudimentary such as"echo 'hi'","true", or even""I still get the error above.This may be noise: We noticed this in 6.14.4 from 6.13.4 - when our Node version transitioned from 12.16.2 to 12.16.3. I was able to walk commits and found the problem reliably appeared at this commit. All it does is points cacache to the
_cacachesubdirectory, whereas beforehand there was no subdirectory. Later we were able to reproduce the error in 6.13.4, so I think the_cacachechange might exacerbate the issue, but is not the direct cause.I made some direct edits to my local copy of npm and found that if, in
pack.js, I changecliArgsto just be[]that everything Just Works(tm). I don't believe this is a sustainable fix though. Further debugged revealed the issue to beE2BIG- an error that a single command line argument exceeded the kernel's restriction. Ours is set to2097152, which I understand to be the Linux default - and is restricted by the kernel's page size. You can view your own withgetconf ARG_MAXat your terminal. I inspected thecliArgsand found thatcawas set to the cert data of every certificate on the machine. I perused these and found a standard assortment of CA certs, intermediate certificates, etc. Our organization's internal certificates were included. I assume but have not yet verified that this is simply reading from the local machine's OpenSSL's certificate list - I know that we don't set it explicitly..npmrchas onlycafile=/etc/pki/tls/certs/ca-bundle.crtin relation to certificates. The build machine where this behavior is observed is running CentOS 7.7.1908. Most of us use Macs at the workplace, and we do not see this error on them with the same node/npm versions.Since this restriction applies to a single argument, I chopped up
cainto multipleca[]arguments. This seems to work forpack.jsbut it kicks the can down the road. I suspect that other parts of npm need a similar treatment, but the processes have either become too opaque for me to trace much further, or I my simian mind is too simple and unfamiliar with npm innards.