Explicitly add stream to cache on `cache add` by mjsir911 · Pull Request #2976 · npm/cli

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 · 2021-03-27T22:48:10Z

Pacote doesn't do this automatically anymore

Fixesnpm/pacote#73 & Fixes#2160

isaacs · 2021-03-31T21:30:44Z

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

mjsir911 · 2021-03-31T22:37:25Z

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 · 2021-04-01T01:24:52Z

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Not sure the implications of changing the return type of _istream / adding a `cacache` to the end of the pipe, too unfamiliar with streams Fixesnpm#73 & Fixesnpm/cli#2160 Also see npm/cli#2976 for my first attempt & some discussions as to how we got here.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

@mjsir911

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976 PR-URL: #74 Credit: @mjsir911Close: #74 Reviewed-by: @isaacs EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes.

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@isaacs

Tell `fetch` not to cache it's downloaded tarballs, leave that to us Also see npm/cli#2976 for my first attempt & some discussions as to how we got here. Fix: #73Fix: npm/cli#2160Close: npm/cli#2976Close: #74 EDIT(@isaacs): Updated to add test, allow make-fetch-happen to use the cache, pipe into cache properly, and avoid cache double-writes. PR-URL: #75 Credit: @mjsir911, @isaacsClose: #75 Reviewed-by: @wraithgar, @mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org>

@mjsir911

This is a backwards incompatible change to the undocumented `cache add pkg version`, but Motivations for this can be found here: npm#2976 (comment) Signed-off-by: Marco Sirabella <marco@sirabella.org> PR-URL: npm#3098 Credit: @mjsir911Close: npm#3098 Reviewed-by: @wraithgar

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

mjsir911 requested a review from a team as a code owner March 27, 2021 22:48

mjsir911 force-pushed the msirabella/fix_cache branch 4 times, most recently from d93af78 to 41a60c8Compare March 28, 2021 00:30

mjsir911 changed the title ~~Explicitly add stream to cache~~Explicitly add stream to cache on cache addMar 28, 2021

darcyclarke added Release 7.x work is associated with a specific npm 7 release semver:patch semver patch level for changes Needs Review labels Mar 29, 2021

darcyclarke assigned isaacs Mar 31, 2021

darcyclarke added the release: next These items should be addressed in the next release label Mar 31, 2021

wraithgar removed the release: next These items should be addressed in the next release label Mar 31, 2021

mjsir911 force-pushed the msirabella/fix_cache branch from 41a60c8 to 887fdf0Compare April 1, 2021 01:21

Explicitly add stream to cache
0e487d2
Pacote doesn't do this automatically anymore Closesnpm/pacote#73 & Closesnpm#2160

mjsir911 force-pushed the msirabella/fix_cache branch from 887fdf0 to 0e487d2Compare April 1, 2021 01:23

darcyclarke closed this Apr 1, 2021

mjsir911 mentioned this pull request Apr 1, 2021
Cache tarballs streamed through pacote npm/pacote#74
Closed

mjsir911 mentioned this pull request Apr 18, 2021
feat(cache): accept multiple specs #3098
Merged

Uh oh!

Conversation

mjsir911 commented Mar 27, 2021

Uh oh!

isaacs commented Mar 31, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjsir911 commented Mar 31, 2021 via email

Uh oh!

mjsir911 commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mjsir911 commented Mar 27, 2021

Uh oh!

isaacs commented Mar 31, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjsir911 commented Mar 31, 2021 via email

Uh oh!

mjsir911 commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mjsir911 commented Mar 27, 2021

Uh oh!

isaacs commented Mar 31, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjsir911 commented Mar 31, 2021 via email

Uh oh!

mjsir911 commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mjsir911 commented Mar 27, 2021

Uh oh!

isaacs commented Mar 31, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjsir911 commented Mar 31, 2021 via email

Uh oh!

mjsir911 commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mjsir911 commented Mar 27, 2021

Uh oh!

isaacs commented Mar 31, 2021•
edited
Loading

isaacs commented Mar 31, 2021•
edited
Loading

isaacs commented Mar 31, 2021•
edited
Loading

isaacs commented Mar 31, 2021•
edited
Loading

isaacs commented Mar 31, 2021•
edited
Loading

isaacs commented Mar 31, 2021•
edited
Loading