Maintenance of the bundled package manager

[jacksonrayhamilton]

On two occasions I’ve dedicated my time to try to improve the package manager that ships with Node, and both experiences have been poor:

• Log failed optional platform dependencies as info. npm/npm#19198
• Stabilize optional shrinkwrapped dependencies. npm/npm#19233

In the first case, due to a hot issue not being referenced/closed by a PR that addressed it, and me not knowing of that PR, the time I dedicated ended up being spent in vain. This was a little bit my fault for assuming that the problem still needed fixing, since the issue was still open and there was no announcement from the maintainers there that it’d been fixed. Still, better communication on the maintainers’ part would have helped avoid that scenario too.

In the second case, the community has been yearning for years for a change to a confusing logging behavior when installing fsevents on Linux. Four PRs in total have been submitted to address this issue:

• install: log failed optional dependencies as info instead of warn (fixes #11632) npm/npm#12841
• install: use log.info for incompatible optional deps npm/npm#17530
• Log failed optional platform dependencies as info. npm/npm#19198
• Log failed optional platform dependencies as info npm/cli#169

8 months went by after I submitted my PR (the third one) before it was auto-closed, and another 9 months have gone by since another contributor has re-submitted my PR (the fourth one). In all that time and with much campaigning in the way of upvotes and comments (not to mention the hundreds from the addressed issue) we’ve been unable to attract sufficient attention from maintainers to get the patch reviewed and merged. This is a +42 −29 patch with tests.

NPM’s CLI, accepting pull requests, appeared as if it would welcome my generous contribution dedicated to Node users’ greater good. Instead, my contribution has been met with much indifference, which I believe to be unwarranted.

The pattern I’ve noticed in both of my experiences has been a lack of concern by npm Inc to communicate with its community in the pursuit of addressing issues together. It’s very discouraging that I have the will to improve this vital component of Node’s ecosystem, but that my labor goes largely unrecognized by the component’s maintainer.

Seeing as the Node.js TSC “is responsible for … a number of projects depended upon by Node.js Core,” I expect this will be a fitting forum to consider the efficacy of npm Inc’s stewardship of the package manager included in Node installations.

Are my observations of neglect isolated, or is there a greater set of cases broad enough to warrant some upheaval? If so, a serious discussion about the management of npm/cli with npm Inc would be warranted. (Can process improvements be implemented for triaging PRs? Can community members be recruited and granted privilege to help?) If management could not be improved, then perhaps stewardship of the CLI ought to pass to Node.js itself. Let this be the beginning of a discussion that may lead to solutions such as these.

[othiym23]

Full disclosure: I was once employed by npm, Inc., and my duties included managing the CLI team, heading up community engagement with the CLI user base, and acting as a coordinator between npm and the Node.js Foundation. I am presently a member of the Node.js moderation team, although I don't believe it to be appropriate for me to participate in moderation of this discussion as I have an interest in it (and am now participating in it).

I am the furthest thing from a disinterested bystander, but it has been several years since I left npm, and I believe I have some distance and perspective now to match my previous level of investment. To be clear, there are points in the linked threads where you, @jacksonrayhamilton, mention me, but those points happened after I was no longer at npm, and the nature of my departure was such that it didn't make sense for me to continue to participate in the project in any form.

With that said, I don't believe this is a good forum for this discussion. While npm is bundled in official Node.js distributions, and while the Node.js project has a duty of care to its users that it be fit for purpose which extends to the things that it bundles, npm is in all other ways an independent project with its own governance, roadmap, priorities, and problems. I believe that the goal of the team working on the CLI is the same that it has always been, which is to make sure that the users of the npm CLI have as smooth and unsurprising experience when installing JavaScript packages as is practicable. I also believe that the size of the team (still very small) is vastly outstripped by the scale and level of expectation on the part of its user community (huge).

npm has always had unusually transparent governance. Even with all the tumult surrounding npm as a company over the last year or so, there have been several blog posts outlining the roadmap that the team is working from. npm has always had some kind of dedicated community site (whether that's Discourse or a GitHub issue tracker) for discussions, and its various maintainers have always been easily reachable on a wide variety of channels (occasionally too much so – I'm not the only npm maintainer to have been pushed deep into the red of burnout by trying to remain engaged and accessible to the broader JS community). It has a well-defined RFC process. It has a constellation of contributors who have given freely of their time and energy simply out of a desire to support the community of which they are a part.

However, as you have experienced, there are few guarantees for open source projects, even those backed by companies with a major stake in the outcome. As I mentioned above, the team is small, and time is short. Open source maintainers, particularly those of popular projects, never have enough time. Time to fix problems, but also time to share context and bring clarity to people as to why the project may or may not be adopting certain solutions or merging certain changes.

I think this is just reality, and not something that can be remediated by replacing one form of governance with another. Even if many more maintainers are added and more efficient processes are adopted to minimize overhead and ensure the timely consideration of proposed changes (or whatever weaknesses a given set of people feel should / must be addressed), somebody or some group of people still must exercise judgment and decide which changes are going in, and which aren't. And it is going to be difficult, if not impossible, for the maintainers to be able to explain their reasoning to everyone involved satisfactorily. Just keeping the team all facing the same direction is hard enough (management complexity grows combinatorily as the number of developers grows linearly), much less somehow finding the wherewithal to keep the thousands of external users aware of what the team's priorities are and why.

I'll put it plainly: there will always be situations in a project like npm where the users feel like the maintainers have dropped the ball. Sometimes that will be true, but more times there will be more going on than is immediately visible. As a potential contributor to a project with millions of other users invested in the tool meeting their own needs, you are not always going to get what you want, even if it seems like what you want is a small, self-evidently reasonable change. This is not something that can be fixed by changing who's responsible for making the decisions or reworking the process by which decisions get made; it's simply a consequence of popularity and living in a world of all-too-finite constraints. It sucks that you feel like your time and contribution weren't properly honored, but, even not being involved with npm in any capacity anymore, I feel absolutely confident in saying that there's no malice directed at you or any of the other participants in the threads to which you link by anyone on the project.

[jacksonrayhamilton]

Thank you for your perspective, @othiym23.

I don't believe this is a good forum for this discussion.

My reasoning for starting this conversation here was the catch-22 I face: I didn’t expect to get a response in the NPM repo (at least for a long while), given the slow/absent communication I experienced and wanted to address. That, and I figured I may find people here who can talk to people there (or who could advise me well on how to do that, as you have done—thank you for that).

I also believe that the size of the team (still very small) is vastly outstripped by the scale and level of expectation on the part of its user community (huge).

This may be the root of the problem, and it ought to be solvable. It may be necessary to bring or let more people in. It could also be necessary for stewardship of the package manager to shift before that would happen.

You seem defeated by the prospect of scaling an open source operation. I don’t think such defeatism is productive. Organizations can scale, so management of Node.js’s bundled package manager can too.

If npm Inc is mired in an impenetrable fog of bureaucracy or too strapped for cash to serve the community, then those are also problems that could potentially be solved by reallocation of community resources. (When I say “community resources,” I am not speaking euphemistically of the Node.js org; I am speaking more broadly, thinking of volunteers, and considering the possibility of corporate donations by those who depend on NPM.)

And since I invoked the image of Microsoft’s checkbook, I must humbly disclaim that I do not consider my particular pull requests worthy of such patronage. I merely took a small sample of bad service from a big package, and I am now polling in this thread to gauge whether the sum total of the community’s grievances would be well served by a degree of change in NPM’s management processes and/or personnel.

[sam-github]

The hope that nodejs would be doing a better job of maintaining npm then npm, inc seems forlorn to me. They have a (small) but full-time team devoted to its maintenance, and work in a blizzard of attention, I personally wouldn't be eager for nodejs to take that over, we don't have an army of volunteers ready to step up and maintain it, and more particularly, no structure for roadmapping, triage, assigning of people, coordination of all those (hoped for) volunteers. Also, there is already an alternative to npm, yarn, so anyone thinking that a package install client would be better with a different managment structure has a clear alternative project to test that theory with. The idea that the nodejs org could somehow do better than both those groups of maintainers doesn't seem credible to me.

[othiym23]

I have a couple brief responses to a few specific points you raise, and then I'll bow out of this discussion.

You seem defeated by the prospect of scaling an open source operation.

If npm Inc is mired in an impenetrable fog of bureaucracy or too strapped for cash to serve the community, then those are also problems that could potentially be solved by reallocation of community resources. (When I say “community resources,” I am not speaking euphemistically of the Node.js org; I am speaking more broadly, thinking of volunteers, and considering the possibility of corporate donations by those who depend on NPM.)

Consider the shading you gave this discussion, because there's a negativity and imputation of bad faith in what is quoted above that is not coming from what I wrote. If you go back and read what I said, I don't think you will find defeatism, nor a sense that the problem is one of bureaucracy.

I don't believe that the problems of managing a popular open source project are, in the end, problems of scale or the correct allocation of resources. The biggest problems are of maintaining consensus and pursuing the correct objectives. Maintaining an orientation is the hardest part of management, and this is especially true when you have a team composed of volunteers who may have very little common purpose aside from solving their own immediate problems or a broad sense of serving the community surrounding a project.

Each additional stakeholder added to a project brings with them their own goals and perspectives. Establishing and preserving consensus grows increasingly untenable as the number of participants grows, and in the absence of consensus, there must be some other means of coming to decisions if a project is to continue making progress. There will always be a core group of people who make the important decisions, and they will inevitably make decisions that displease other stakeholders. This is what it means to not have consensus, especially in the black and white world of OSS discourse, where "disagree and commit" is not really a thing.

I merely took a small sample of bad service from a big package

Your frustration is plain, so in return I will be equally blunt: npm owes you no service. npm's maintainers owe you nothing at all. You offered your contribution, and it was not taken up – any additional interpretation of what happened is simply interpretation, not a record based on actions. There is no contract or obligation, explicit or implied, governing what npm will do with community contributions, and while it would be friendly and helpful if you were given explicit feedback on why things played out the way they did, that is not something to which you, or any other participant in the community, are entitled.

[jacksonrayhamilton]

@sam-github thank you for sharing your perspective on this; it is useful to know that Node.js’s participation may be untenable.

The idea that the nodejs org could somehow do better than both those groups of maintainers doesn't seem credible to me.

If, theoretically, npm Inc were only able to respond to PRs at a rate of 10% satisfaction, but a different maintainer could handle them at a rate of 20%, then that would be an improvement. Part of the purpose of this thread is to gauge if matters are really that bad, or if my case is just isolated.

@othiym23,

I don’t understand why you consider consensus problematic. I am imagining several different maintenance configurations:

1. A single developer is responsible for reviewing and accepting all PRs for a project
2. 3 developers are responsible for PRs concerning their respective thirds of a project
3. 6 developers in teams of two are responsible for PRs concerning their respective thirds of a project
4. etc…

Wouldn’t each multiplication of manpower result in better results than the one prior? (Yeah, maybe it’s not presently available, but is it needed? Should it be sought after for the greater good?)

There will always be a core group of people who make the important decisions, and they will inevitably make decisions that displease other stakeholders.

Why? Why not bring in more people to make more decisions, and let them?

npm owes you no service.

In an ideal, healthy open source community, they do. By creating a repository on GitHub and allowing PRs to be opened on it, they have created the expectation that the submission of a good PR that they find valuable will eventually be rewarded by the PR being merged. But in the case of npm/npm#19198 / npm/cli#169, instead a cruel trick is played on us; a carrot is dangled forever beyond the community’s reach.

Imagine a lot designated a “Community Garden.” There is a sprinkler system there, so neighbors expect flowers they plant there to be watered. But they whither, because it happens that the sprinklers are broken. Do we want to enjoy a blooming Spring in our community garden or not? It would be good if the lot’s owner replaced the sprinklers; but barring that, knock down the sign or let another another chap donate his yard, lest another poor sap on this street plants seeds in the barren field.

[jacksonrayhamilton]

Consider the shading you gave this discussion, because there's a negativity and imputation of bad faith in what is quoted above that is not coming from what I wrote. If you go back and read what I said, I don't think you will find defeatism, nor a sense that the problem is one of bureaucracy.

It seemed so reasonable to me that there are avenues for improvement that I felt it would be patronizing to enumerate them. (e.g. the Roman empire started as a small tribe but grew to be great; therefore, organizations can grow.) That’s why I divined some practical explanations instead.

I left some questions in my last post. I can imagine that the answers to them are “yes;” however, I have little management experience, particularly in regards to large open source projects, so I welcome your firsthand experience in this area. It just didn’t compute for me. Thank you for putting in as much effort as you already have to try to help me understand where you’re coming from.

[darcyclarke]

Hey @jacksonrayhamilton!

My name is Darcy & I actually happen to be the Engineering Manager for Community & Open Source at @npm.

First off, I'm sorry to hear that you've had bad experiences trying to contribute to the npm/cli in the past. I actually recently wrote about some changes we've made that might help eliminate some of the problems you've run up against. That said, I recognize the road ahead to cleaning up and making our projects accessible - to the widest community of contributors possible - will likely take more time, more patience and may never be perfected (as @othiym23 already elaborated on very eloquently).

The good news is, our team has been hard at work to try and improve the contribution, collaboration and visibility story and I'm happy to hear that you share in that passion to grow and foster this community.

Some recent changes we've made to try and improve contribution, collaboration and visibility include: archiving the npm.community forums to help focus support on and reopen GitHub issues for the cli, begun hosting bi-weekly Open RFC meetings that are live streamed and that anyone can join (you actually just missed out on that today), Public Project & Status Boards, promoting and maintaining a Public Events Calendars as well as ramping up our activity/involvement in the Package Maintenance & Tooling Node.js Foundation Working Groups.

As has been the case for awhile, we also actively encourage people to draft RFCs for changes/improvements to the cli or open registry.

Furthermore, we've been working, internally, to reschedule a bi-weekly release cadence which will see the team dedicate time, on alternating weeks, to bubble up issues and PRs ready to be pulled in to the next v6.x release. Along the way, we hope to address the, yet to be pruned, backlog of issues that were carried over from the legacy forums.

All this is to say that I'd definitely love to hear your feedback on any other ways you think we might be able to better support and enable community contributions to our open source projects.

Our entire team is extremely friendly and approachable and there's a number of ways you can reach out to us to hopefully address any concerns going forward. On a personal note, my DMs on Twitter are always open and/or you are free to email me if that would easier: darcy@npmjs.com.

I believe we're just getting started with some amazing work that should hopefully unlock better experiences for the community at large going forward.

~Darcy
❤️

[jacksonrayhamilton]

Hi Darcy, thanks for getting back.

I feel like a petition to get a PR reviewed falls outside the scope of an RFC meeting. Isn’t that for reviewing incubating ideas?

Furthermore, we've been working, internally, to reschedule a bi-weekly release cadence which will see the team dedicate time, on alternating weeks, to bubble up issues and PRs ready to be pulled in to the next v6.x release.

That would be an avenue for noticing PRs wanting review/merging.

All this is to say that I'd definitely love to hear your feedback on any other ways you think we might be able to better support and enable community contributions to our open source projects.

I’m glad to see a pull request template has been added with a “references” heading. If used consistently, that could prevent foibles like the one I described “in the first case” in the OP. It may also be helpful for the template to encourage the contributor, something to the effect:

“Please do a quick search to see if this closes any existing issues. Many people may be thankful for your contribution and this will let them know about it!”

Also, an idea I just came up with:

A scale could be decided upon to help with prioritizing the processing of PRs. e.g.: “high,” “medium” and “low” review priority. This could be based on community feedback (issue got 100+ upvotes = high; 25+ upvotes = medium), or simply arbitrary judgment. PRs without ratings are rated on a weekly basis. PRs with high priority get reviewed by the end of the week they’re submitted. PRs with medium priority can be reviewed up to 2 weeks since submission; for low ones, up to a month since. This way the biggest impact stuff gets to users quickest, and all contributions get roadmapped.

[mhdawson]

@jacksonrayhamilton wondering if there is a next step on this issue ? I see there was some discussion and @darcyclarke jumped in from the npm side. Do you still think the same problem exists and have ideas of what the TSC should consider doing?

[jacksonrayhamilton]

@mhdawson as far as I know, the problem still exists.

The problem is that NPM is knotted into Node and, in my experiences, the company charged with maintaining NPM doesn’t act like it cares about community contributions, nor the people who provide them. Thus, Node’s overall ability to be improved by its community is limited by npm Inc.

Essentially, it took over four years for a community-submitted change (npm/npm#12841, npm/npm#17530, npm/npm#19198, npm/cli#169) to finally be obsoleted by NPM’s own fix for the problem. In less than three years (there was some feedback in the first year, but not in the last three), in exchange for the community’s generous contributions, I expected npm Inc to provide some conclusion to the issue like:

“Thanks for these changes, we’re going to release them.”
or: “These changes are unacceptable, we’re closing this.”
or: “Can you adjust these changes so they are acceptable?”
or: “We’re going to handle this differently ourselves.”

Apparently, sometime throughout this ordeal, npm Inc was aware of the problem, because they announced they were fixing (or had already fixed?) the problem in a bullet point in a blog post.

npm Inc could have indicated its concern for community contributions by responding to GitHub issues and closing obsolete PRs as soon as it began implementing its own fix for the issue. I don’t know when they decided to fix the issue themselves, but depending on how long ago they did, simply by responding they could have saved the community wasted energy (months, maybe years) campaigning for an obsolete patch.

Since Node decides what package manager code is bundled in its distribution, in the interest of granting its community the privilege of actually receiving responses to community contributions, the TSC could consider assuming (from npm Inc) the responsibility of maintaining Node’s bundled package manager.

That is my perspective and proposition. You can weigh that against the other reasonable opinions in this thread.

Ideally, npm Inc would carve out some time in its employees’ schedules to respond to threads on GitHub, so contributors like me would not be totally alienated from the prospect of providing their company free code and for the common good of our ecosystem. If npm Inc fulfilled that responsibility and obligation to the commune which is their FOSS project, then no heavy-handed redistribution of that responsibility and obligation would be necessary to resolve this problem.

[ljharb]

npm accepts community contributions all the time - "not merging PRs" is a normal part of open source maintenance, not a sign of neglect. Even expecting responsiveness on a product you don't pay for - which includes both node or npm - borders on unreasonable, although it's certainly the sort of courtesy one might hope for.

I personally have over 100 unread Github notifications at any one time, and none of the projects I maintain have the reach or userbase of npm itself - so despite my best effort, I'm sure there's plenty of issues and PRs that I have simply seemed to ignore, but in fact, I haven't had time to address.

[MylesBorins]

@jacksonrayhamilton howdy o/ while I am a Node.js TSC member I am also a product manager at GitHub working on npm (if you missed the news GitHub acquired npm in march)

There has been a ton of heads down work on npm 7, which we are hoping is going to be much easier to maintain in the long term... and are excited that we should have it out the door in time for Node.js 15. Darcy mentioned earlier in the thread the work we've done around the RFC program to help make it easier to get directly involved in the direction of npm, and we have some other stuff we are working on that should be available soon to make it even easier to get us actionable feedback.

The team working directly on npm is not nearly as large as you might believe it to be, but I can assure you they are going to be paying close attention to the issue tracker as they work to improve reliability, performance, testing, and documentation of npm 7.

There is active work happening to expand the number of available package managers out of the box in Node.js core, that work may interest you as well.

While I can't do anything about issues from before my time with GitHub (I only joined in June) I can offer to be a point of contact if you run into issues in the future, my DMs are open and my email address is available in the README.md in nodejs/node.

[jacksonrayhamilton]

Thanks for reaching out @MylesBorins. That’s all great to hear and I will keep it in mind.