Provide visibility into dependency tree permutations

[achrinza]

With the exception of the LoopBack CLI, the Node.js packages under the purview of the LoopBack project are commonly used as libraries (i.e. as a dependency of another Node.js project). Due to the nature of transitive dependencies, every dependency tree generated by LoopBack users - even if its the same package version - may differ significantly as new versions of direct and deep dependencies are released. This makes it difficult for the maintainers of LoopBack to monitor the impact of another Node.js package on LoopBack users, as it is unclear whether that dependency may have been installed as some kind of deep dependency of a LoopBack package.

While LoopBack users do have the tools necessary to check if their specific installation is compromised, not all users may have the means or time to check the impact of these issues. Hence, this issue is to track how the LoopBack project can reduce the effort needed to provide a brief and accurate insight into the impact of a Node.js package vulnerability to users of LoopBack packages.

Although it is not expected that the LoopBack project do this work for its users, it would be a plus-point if we can streamline this process and provide actionable insights for operators who want a more hands-off experience when managing LoopBack-based projects.

This issue is to track the brainstorming of ideas and solutions toward solving the aforementioned problem statement.

Research done thus far

The technical problem at hand is that there's no easy way to find out every possible Node.js package that may be installed by users of LoopBack packages.

The goal is to reduce the user's initial analysis to a simple glance at their package.json direct dependencies.

This problem isn't easily solved with readily-available APIs or solutions as:

1. It requires heavy traversing of every permutation of the dependency tree, and
2. significant calls to the NPM public registry to retrieve package metadata

One way to work around this heavy traversal requirement is to host a read-only replica of the NPM public registry, as suggested by https://docs.npmjs.com/policies/crawlers (emphasis mine):

if you want a full copy of our metadata it is always going to be faster to access the data via replication, which takes only an hour or two to provide full data and will thereafter automatically stay in sync.

Behind the scenes, the NPM public registry is powered by CouchDB, and its database ~95 GB in size according to https://replicate.npmjs.com/:

"sizes": {
 "file": 95630138970
}

This means that it may be feasible to host a copy of the NPM public registry, and perform queries against it to build the dependency tree permutations.

Replication API documentation: https://github.com/npm/registry/blob/9e368cf6aaca608da5b2c378c0d53f475298b916/docs/REPLICATE-API.md

Further areas of exploration:

• Logic for getting dependency tree permutations for a specific package version (e.g. @loopback/core@1.0.0)
• File format for storing this information
• Automated and optimised periodic updating of dependency tree permutations
• Corrolating and detecting existence of vulnerable dependency