Skip to content

http2: track memory allocated by nghttp2 - #21374

Closed
addaleax wants to merge 1 commit into
nodejs:masterfrom
addaleax:track-memory-http2
Closed

http2: track memory allocated by nghttp2#21374
addaleax wants to merge 1 commit into
nodejs:masterfrom
addaleax:track-memory-http2

Conversation

@addaleax

Copy link
Copy Markdown
Member

Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.

This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.

It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.

Refs: #21373
Refs: #21336

[blocked by both of these fixes]

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows commit guidelines

@addaleaxaddaleax added blocked PRs that are blocked by other issues or PRs. http2 Issues or PRs related to the http2 subsystem. labels Jun 16, 2018
@addaleax
addaleaxforce-pushed the track-memory-http2 branch from f8aa056 to 922f2ecCompareJune 16, 2018 22:09
@jasnell

Copy link
Copy Markdown
Member

Woo!

@trivikrtrivikr added the c++ Issues and PRs that require attention from people who are familiar with C++. label Jun 17, 2018
Comment threadsrc/node_http2.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to change the hpack headers part, as it is no longer visible in code?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, done!

@addaleax
addaleaxforce-pushed the track-memory-http2 branch from 922f2ec to 50a9d42CompareJune 20, 2018 20:54
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
Refs: nodejs#21373
Refs: nodejs#21336
@addaleax
addaleaxforce-pushed the track-memory-http2 branch from 50a9d42 to fa8d967CompareJune 20, 2018 20:54
@addaleaxaddaleax removed the blocked PRs that are blocked by other issues or PRs. label Jun 20, 2018
@addaleax

Copy link
Copy Markdown
MemberAuthor

Rebased, this should no longer be blocked

CI: https://ci.nodejs.org/job/node-test-pull-request/15544/

@addaleaxaddaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jun 20, 2018

@TimothyGuTimothyGu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still LGTM.

Comment threadsrc/node_http2.cc
if (mem != nullptr) {
// Adjust the memory info counter.
session->current_nghttp2_memory_ += size - previous_size;
*reinterpret_cast<size_t*>(mem) = size;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It took me a while to realize how this would behave for a size of 0: our UncheckedRealloc implementation will return nullptr in that case, to fill in an implementation-defined gap in the system realloc(). It might be helpful to clarify that.

@ChALkeRChALkeR added the memory Issues and PRs related to the memory management or memory footprint. label Jun 22, 2018
@apapirovski

Copy link
Copy Markdown
Contributor

Landed in 15c627f

apapirovski pushed a commit that referenced this pull request Jun 25, 2018
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
PR-URL: #21374
Refs: #21373
Refs: #21336
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
targos pushed a commit that referenced this pull request Jun 25, 2018
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
PR-URL: #21374
Refs: #21373
Refs: #21336
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
@targostargos mentioned this pull request Jul 3, 2018
kjin pushed a commit to kjin/node that referenced this pull request Aug 23, 2018
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
PR-URL: nodejs#21374
Refs: nodejs#21373
Refs: nodejs#21336
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
kjin pushed a commit to kjin/node that referenced this pull request Sep 25, 2018
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
PR-URL: nodejs#21374
Refs: nodejs#21373
Refs: nodejs#21336
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
kjin pushed a commit to kjin/node that referenced this pull request Oct 16, 2018
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
PR-URL: nodejs#21374
Refs: nodejs#21373
Refs: nodejs#21336
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
BethGriggs pushed a commit that referenced this pull request Oct 17, 2018
Provide a custom memory allocator for nghttp2, and track
memory allocated by the library with it.
This makes the used-memory-per-session estimate more
accurate, and allows us to track memory leaks either
in nghttp2 itself or, more likely, through faulty
usage on our end.
It also allows us to make the per-session memory limit
more accurate in the future; currently, we are not
handling this in an ideal way, and instead let nghttp2
allocate what it wants, even if that goes over our limit.
Backport-PR-URL: #22850
PR-URL: #21374
Refs: #21373
Refs: #21336
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs that have at least one approval, no pending requests for changes, and a CI started.c++Issues and PRs that require attention from people who are familiar with C++.http2Issues or PRs related to the http2 subsystem.memoryIssues and PRs related to the memory management or memory footprint.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@addaleax@jasnell@apapirovski@TimothyGu@ChALkeR@trivikr