stream: fix memory usage regression in writable by orgads · Pull Request #53188 · nodejs/node

orgads · 2024-05-28T16:27:38Z

Setting writecb and afterWriteTickInfo to null did not clear the value in the state object.

Amends 35ec931 (stream: writable state bitmap).

Fixes#52228.

nodejs-github-bot · 2024-05-28T16:27:44Z

Review requested:

@nodejs/streams

orgads · 2024-05-28T16:29:54Z

@ronag @mcollina

I'm not sure how to add a test for this. Hints will be welcome :)

I tested with the example app from #52228 and it seems to solve the issue.

orgads · 2024-05-28T18:04:47Z

Actually the condition is equivalent. I'll need to dig deeper.

orgads · 2024-05-28T19:50:59Z

@ronag I tried to inspect that change, but couldn't find a smoking gun. #52228 is definitely caused by it, I bisected and triple-checked. Can you try to find the culprit?

I'm closing this PR, as it is wrong anyway.

orgads · 2024-05-29T05:55:39Z

Ok, I found the culprit. Ready for review.

Can you suggest how to add a test?

benjamingr · 2024-05-29T09:42:50Z

Can you suggest how to add a test?

Good question, I'm honestly not so sure - you can test writecb is set to null in onwrite but I'm not sure that tests something useful. We don't have memory benchmarks I'm aware of though that might be a good idea overall.

mcollina

lgtm

However this is not a memory leak, because the memory consumption is stable.
I would recommend we change the commit/PR title to "stream: fix memory usage regression in writable".

mcollina · 2024-05-29T13:24:45Z

A test for this would be nice but I'm at a loss on how to write it.

nodejs-github-bot · 2024-05-29T13:35:01Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59517/

orgads · 2024-05-29T14:34:04Z

However this is not a memory leak, because the memory consumption is stable. I would recommend we change the commit/PR title to "stream: fix memory usage regression in writable".

Done.

nodejs-github-bot · 2024-05-29T18:48:36Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59527/

Setting writecb and afterWriteTickInfo to null did not clear the value in the state object. Amends 35ec931 (stream: writable state bitmap). Fixesnodejs#52228.

nodejs-github-bot · 2024-05-30T18:17:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59547/

orgads · 2024-05-30T20:50:17Z

smartos build fails, unrelated to my changes.

nodejs-github-bot · 2024-05-31T14:00:57Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59567/

nodejs-github-bot · 2024-05-31T23:38:14Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59588/

nodejs-github-bot · 2024-06-01T05:59:31Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59590/

nodejs-github-bot · 2024-06-01T23:29:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59609/

nodejs-github-bot · 2024-06-02T00:35:37Z

Landed in 78a326e

orgads · 2024-06-02T01:42:46Z

Thanks! What's the process of backporting? Should I cherry pick and create a new PR for 22 and 20?

H4ad · 2024-06-02T01:59:17Z

@orgads Since we didn't put labels like "do not land", this PR is automatically eligible to be included in older versions, just wait for the next version, the releasers will include this PR.

You only need to create backport when releasers find a conflict that doesn't allow them merge it easily.

/cc @nodejs/releasers Just to let you know guys an important fix for memory usage on streams was merged.

Setting writecb and afterWriteTickInfo to null did not clear the value in the state object. Amends 35ec931 (stream: writable state bitmap). Fixes#52228. PR-URL: #53188 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>

Setting writecb and afterWriteTickInfo to null did not clear the value in the state object. Amends 35ec931 (stream: writable state bitmap). Fixesnodejs#52228. PR-URL: nodejs#53188 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>

orgads · 2024-06-21T06:20:09Z

@H4ad @marco-ippolito It hasn't been backported to 20.x. :(

marco-ippolito · 2024-06-21T06:21:50Z

@H4ad @marco-ippolito It hasn't been backported to 20.x. :(

It has not been on the Current enough (2 weeks) to be backported to v20

orgads · 2024-06-21T06:23:11Z

I see. Thanks for the explanation.

Setting writecb and afterWriteTickInfo to null did not clear the value in the state object. Amends 35ec931 (stream: writable state bitmap). Fixes#52228. PR-URL: #53188 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>

nodejs-github-bot added the needs-ci PRs that need a full CI run. label May 28, 2024

benjamingr approved these changes May 28, 2024
View reviewed changes

orgads commented May 28, 2024
View reviewed changes

Comment threadlib/internal/streams/writable.js Outdated

orgads closed this May 28, 2024

orgads reopened this May 29, 2024

orgads force-pushed the writable-temp-leak branch from eb34ffd to 7b9f16bCompare May 29, 2024 05:53

orgads changed the title ~~writable: Fix condition mis-translation~~stream: fix memory leaks in writableMay 29, 2024

orgads force-pushed the writable-temp-leak branch from 7b9f16b to 24b2499Compare May 29, 2024 05:54

orgads mentioned this pull request May 29, 2024
Significant memory usage increase starting from 20.10.0 for a simple HTTP server (potentially stream related) #52228
Closed

benjamingr approved these changes May 29, 2024
View reviewed changes

mcollina approved these changes May 29, 2024
View reviewed changes

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label May 29, 2024

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label May 29, 2024

orgads force-pushed the writable-temp-leak branch from 24b2499 to aec133fCompare May 29, 2024 14:33

orgads changed the title ~~stream: fix memory leaks in writable~~stream: fix memory usage regression in writableMay 29, 2024

lpinca added the request-ci Add this label to start a Jenkins CI on a PR. label May 29, 2024

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label May 29, 2024

stream: fix memory usage regression in writable
61da0ed
Setting writecb and afterWriteTickInfo to null did not clear the value in the state object. Amends 35ec931 (stream: writable state bitmap). Fixesnodejs#52228.

orgads force-pushed the writable-temp-leak branch from aec133f to 61da0edCompare May 30, 2024 09:55

lpinca approved these changes May 30, 2024
View reviewed changes

lpinca added the request-ci Add this label to start a Jenkins CI on a PR. label May 30, 2024

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label May 30, 2024

H4ad approved these changes May 31, 2024
View reviewed changes

H4ad added the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 2, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 2, 2024

nodejs-github-bot merged commit 78a326e into nodejs:mainJun 2, 2024

orgads deleted the writable-temp-leak branch June 2, 2024 01:38

RafaelGSS mentioned this pull request Jun 7, 2024
v22.3.0 proposal #53379
Merged

marco-ippolito mentioned this pull request Jul 19, 2024
v20.16.0 proposal #53945
Merged

Uh oh!

Uh oh!

Conversation

orgads commented May 28, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented May 28, 2024

Uh oh!

orgads commented May 28, 2024

Uh oh!

orgads commented May 28, 2024

Uh oh!

Uh oh!

orgads commented May 28, 2024

Uh oh!

orgads commented May 29, 2024

Uh oh!

benjamingr commented May 29, 2024

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

mcollina commented May 29, 2024

Uh oh!

nodejs-github-bot commented May 29, 2024

Uh oh!

orgads commented May 29, 2024

Uh oh!

nodejs-github-bot commented May 29, 2024

Uh oh!

nodejs-github-bot commented May 30, 2024

Uh oh!

orgads commented May 30, 2024

Uh oh!

nodejs-github-bot commented May 31, 2024

Uh oh!

nodejs-github-bot commented May 31, 2024

Uh oh!

nodejs-github-bot commented Jun 1, 2024

Uh oh!

nodejs-github-bot commented Jun 1, 2024

Uh oh!

nodejs-github-bot commented Jun 2, 2024

Uh oh!

orgads commented Jun 2, 2024

Uh oh!

H4ad commented Jun 2, 2024

Uh oh!

orgads commented Jun 21, 2024

Uh oh!

marco-ippolito commented Jun 21, 2024

Uh oh!

orgads commented Jun 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

orgads commented May 28, 2024•
edited
Loading