dedupe: get deps from shrinkwrap

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?

[larsgw]

See https://npm.community/t/3807

[larsgw]

(Breaking) side effects include:

• installing any package that is in package-lock.json but not in node_modules

It doesn't install packages that are only specified in package.json.

[aeschright]

We're going to make sure this doesn't have any unintended consequences and goes in the right direction.

[aeschright]

We'd like you to redo the test using our standard test script: see scripts/maketest -- you can give it a directory fixture and it will give you the test setup. It'll help us keep things consistent and avoid some older conventions that aren't effective.

[larsgw]

I added the tests, and a doc change now too. I think this change is going to be pretty controversial, seeing how many people already disagree with npm install behavior, fixing incorrect package-lock.json files. The only middle way I can think of, fixing the bug and keeping the behavior, would be to somehow merge all movements in the node_modules tree with all the modules left behind in the shrinkwrap tree, but I don't know if that's feasible.

[G-Rath]

What's the status of this? The community issue was closed, so I can't ask on that.

For me as a Windows user, with my expectation of what ddp does, I find it odd that running ddp results in a package-lock.json that will change if I then run npm i.

The core question I have is: Why is npm i able to generate a package-lock.json that includes fsevents without actually installing that package?

[FranklinYu]

@G-Rath I don’t think that’s an issue. Optional dependency should also be locked, so that when the package is installed on another platform, it won’t change the lock file; this way the lock file converges. In contrary, it doesn’t make sense to remove fsevents when it doesn’t apply to current platform.

[G-Rath]

@FranklinYu For sure.

My question wasn't a challenge at npm i, but as a starting point to try and figure out how ddp should act, as this PR has the breaking side effect of :

installing any package that is in package-lock.json but not in node_modules

The point I was wanting to draw attention to with my question is that npm idoesn't have to install all packages to have them in the package-lock.json, and so why does ddp have to?

While I'm sure there's a good reason, in my eyes that highlights a starting point for identifying where ddp drifts away from i.

[FranklinYu]

installing any package that is in package-lock.json but not in node_modules

I think that only refers to packages that work for current platform. For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

[G-Rath]

For example, since fsevents doesn't work in Windows, even if it's in lock file, not in node_modules, it won't be installed by dedupe.

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Currently, if I run npm i on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get the same lock file regardless of OS.

Currently, if I run npm ddp on any computer, given the same package.json (and technically a complete freeze of time so that no new versions are released), I'll get a lock file that will differ based on what packages are supported by that OS.

Example:

Given this package.json:

{
"optionalDependencies": {
"fsevents": "^2.0.7"
}
}

Running npm i on Windows, Linux, & OSX gives you the following lock:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Running npm ddp on Windows or Linux gives you:

{
"lockfileVersion": 1
}

Running npm ddp on Mac gives you:

{
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"fsevents": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz",
"integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==",
"optional": true
}
}
}

Note that nothing in node_modules has changed - the only change happening here is in the package-lock.json (aside from npm i on OSX, which of course installs fsevents)

[FranklinYu]

Which is fine, except deduperemovesfsevents from the lock file, which is my whole issue :)

Exactly. The whole point of this PR is that dedupeshould not remove fsevents.

[G-Rath]

The whole point of this PR is that dedupe should not remove fsevents.

Exactly. I've never said otherwise; but it's good we're on the same page 🙂

My question is about the side effect:

installing any package that is in package-lock.json but not in node_modules

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From my understanding, the worst-case bare minimal solution would be to simply run npm i after npm ddp. That is of course highly inefficient, but it's a starting point - from there you can look to strip away unneeded parts from npm i until you're left w/ the minimal code required to "fix" the side effect.

That's a "fix it in post" style approach - which isn't bad; those kind of fixes are often just far more simpler & maintainable than the alternative: try to "fix" the actual root cause of the problem.

This could just be b/c I'm misunderstanding what is meant by "installing any package that is in package-lock.json but not in node_modules", or that just due to the complex nature of npm, that's just how it is, but as I've said: there is at least one way to work around it, that I don't see any major downsides to, given that it seems to give what we all agree is the least astonishing result from running such a command.

Eitherway, I'm just generally interested in understanding more about npm works that means that this is a side effect (w/o spending hours taking it apart line by line*) 😄

*: I mean, I'd love to do this, but sadly don't have the time 😂

[larsgw]

I'm interested in why this side effect occurs, given that npm i doesn't have this side effect?

From what I remember (it's been a while), npm dedupe works like this:

1. Read node_modules/ tree (like npm ls, and not package-lock.json like npm i)
2. Perform deduping on the tree
3. Write the tree to node_modules/ and package-lock.json

Since non-installed optional dependencies have no record in the node_modules/ tree they aren't included in the new package-lock.json. To fix that, I see two possibilities:

1. Read from package-lock.json instead of node_modules. This has the added side-effect of installing (or trying to install) any packages that are in the package lock, but not in node_modules. This is because only reading the package lock can't tell you certain packages shouldn't be installed, just like only reading node_modules can't tell you if certain packages should be installed. However, that seemed like a smaller problem to me.
2. What I mentioned above: somehow track the movements of the deduping process and merge those with the package lock, but leave the rest of the process. Way above my pay-grade as a voluntary contributor (and please don't spend time doing that yourself either).

PS: I think I thought of a third option just now, but I can't think of it anymore so I'll come back to this if the thought does.

[G-Rath]

Ah interesting - thanks for the explanation :)

Would it be viable to try and leverage the logic that npm i uses to determine if a package should be installed or not?

That could be going in the direction you said not spend time trying to do via 2. :)

I'm also wondering if there is any to know (or get an idea of) all the possible conditions that can result in a package not being installed but being written to package-lock.json.

So far I only know of optionalDependencies, which from what happens w/ fsevents, seem to always be included in the package-lock.json, and so wouldn't be something ddp has to worry about?

I feel like this strengthens the argument that ddp should optimise based on package-lock.json: while it's annoying, if you're going to install a dependency in some situations, you should optimise for that situation, since otherwise isn't it harder/more work to safely determine what packages should be installed?

This has the added side-effect of installing (or trying to install)

To me that sounds like it'll work, depending on how the "trying" is handled; Looking at the error output from doing npm i fsevents on Windows, it's throwing EBADPLATFORM - That to me seems like an exception that'd only really be thrown in a predictable situation (unlike say EBADPERMISSIONS^).

Which I think boils back down to the first part of this comment, about how does npm i determine if a package should be installed, and can that be leveraged here?

*: Luckly, fsevents has no dependencies, and from what I've seen, it's the most common optionalDependencies out there.
^: Might not be an actual exception, but you know what I mean

[FranklinYu]

I like option 1. Package lock file should be the source of truth. Everyone knows that node_modules is transient and nobody checks it into version control.

[G-Rath]

Plus if you do "need" to modify anything in node_modules, you can use patch-package to do so stably.

[FranklinYu]

sane seems to have switched from fsevents package to fs.watch(). For future testers, we have other examples:

• lighter-run (only 2 dependencies)
• Chokidar (popular)

Both depend on fsevents for now.

[jpsfs]

@FranklinYu any update on this?

[FranklinYu]

@jpsfs I think you’re mentioning the wrong user. I’m not part of npm team so I can’t merge this even though I want this feature. I haven’t see any npm member involved in discussion by far, so I wouldn’t expect this to happen any time soon.

[darcyclarke]

@larsgw sorry for the delay here. We're going to close this as much of this code has been moved to Arborist (dedupe has been rewritten in v7)

[FranklinYu]

@darcyclarke Did you imply that NPM v7 won’t have this issue any more?