http2: use aliased buffer for perf stats, add stats by jasnell · Pull Request #18020 · nodejs/node

jasnell · 2018-01-06T22:11:24Z

Add an aliased buffer for session and stream statistics, add a few more useful metrics

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

http2

addaleax · 2018-01-06T22:17:25Z

The net & fs APIs use bytesWritten and bytesRead for these values… might be worth being consistent with that?
Edit: Thinking about it, I would have a pretty strong preference for consistency, because that means we might be able to consolidate some of the code around handling C++ streams on the JS side…

addaleax · 2018-01-06T22:17:25Z

Just curious, why not use plain writable properties? Is that what we do in the rest of the perf API, or even spec-mandated?

We can, it's not spec'd to require it.

TimothyGu

Generally looks good.

TimothyGu · 2018-01-08T07:33:04Z

This could probably be merged with the line before.

mcollina · 2018-01-08T14:53:25Z

Can you move this into the http2 namespace? I'm kind of -1 on having perf-hooks require something within http2 itself.

Not following. The collectStats method is in the internal/http2/util.js file. It is used here because we are instrumenting the http2 performance entry object with the http2 specific details, which must be done within this function.

The rest of the code in perf_hooks is isolated from the rest of core. We are not loading fs, http, net etc to instrument them. Why do we need to load a part of http2?

ah, ok, so move the collectStats method in to perf_hooks to keep from having to require anything from http2... yeah, I can do that.

why do we need this for http2 while we don't for http or net?

Neither http or netcurrently output any performance metrics via the perf_hooks API. That's something that's on my lower priority todo list. This is there for http2 because we're using an aliased buffer to move those data values out to js instead of setting properties on the object from within the C++ code -- it's far more efficient to do it this way. Eventually, once other bits of code are instrumented with perf_hooks output, similar mechanisms will be added. For instance, I'm planning on doing something similar with dns lookups fairly soon.

jasnell · 2018-01-08T17:07:53Z

Updated to address the feedback. PTAL

jasnell · 2018-01-08T21:16:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/12447/

jasnell · 2018-01-08T21:41:39Z

CI: https://ci.nodejs.org/job/node-test-pull-request/12448/

Add an aliased buffer for session and stream statistics, add a few more metrics

jasnell · 2018-01-08T22:19:05Z

Trying again with a possible fix for the seemingly unrelated failure: https://ci.nodejs.org/job/node-test-pull-request/12450/

jasnell · 2018-01-08T22:45:42Z

Once again: https://ci.nodejs.org/job/node-test-pull-request/12451/

apapirovski · 2018-01-08T22:49:11Z

I'm reasonably certain this is not a flaky test. I ran it on my local environment against master and it succeeded. I ran a test against this PR and it fails.

jasnell · 2018-01-08T22:50:37Z

yeah, except this PR doesn't touch that code at all, which is quite strange. The only thing I can think of is that there's some weird interaction with AliasedBuffer.

jasnell · 2018-01-08T22:55:13Z

I'm thinking it has something to do with some alignment issue caused by the new aliased buffers. Trying a different fix.

apapirovski · 2018-01-08T23:01:06Z

It's not this PR directly. Just including process.binding('http2') causes this test to fail. So I assume something within the binding initialization is not quite right and it's exposed now because it gets included by perf_hooks.

jasnell · 2018-01-08T23:04:32Z

perf_hooks is not used at all by the failing test

apapirovski · 2018-01-08T23:06:42Z

perf_hooks is not used at all by the failing test

It's indirectly required by bootstrap_node, which requires internal/process which then requires perf_hooks.

Anyway, a simple reproduction on master is:

'use strict';process.binding('http2');constcommon=require('../common');process.on('beforeExit',common.mustCall(()=>{setTimeout(common.mustNotCall(),1).unref();}));

addaleax · 2018-01-08T23:09:05Z

@apapirovski I can’t reproduce with that :/

And btw, this reminds me of 941c65b …

jasnell · 2018-01-08T23:18:49Z

Yeah, that was my first thought too @addaleax but I couldn't find anything that was keeping the loop open. Trying something else... CI: https://ci.nodejs.org/job/node-test-pull-request/12453/

jasnell · 2018-01-08T23:30:52Z

Ok, lazy loading the process.binding('http') in perf_hooks fixes it, so thank you for that @apapirovski. Unfortunately, that still doesn't explain why it's happening, as there shouldn't be side effects from process.binding('http2') that should cause this. I've looked a couple times now and can't see anything. @addaleax, can I ask you to take a look to see if anything jumps out at you?

addaleax · 2018-01-08T23:35:11Z

Yes, but that will have to wait until tomorrow :)

Fishrock123 · 2018-01-08T23:52:24Z

I also cannot reproduce from master the way @apapirovski has.

apapirovski · 2018-01-08T23:53:28Z

@addaleax @Fishrock123 I'm guessing you're both on Windows? I can reproduce on Linux & macOS.

Fishrock123 · 2018-01-08T23:54:18Z

I am currently using macOS 10.12.6.

jasnell · 2018-01-09T00:14:43Z

I'm unable to duplicate on Ubuntu VM or Windows. There really should not be anything in the process.binding('http2') that would keep the loop alive. What could be happening is some weirdness with the http2_state mechanism and the aliased buffers since there is an effect there on env. That's the only real change introduced by this PR so that's the most likely culprit.

Scheduling a PerformanceGCCallback should not keep the loop alive but due to the recent switch to using the native SetImmediate method, it does. Go back to using uv_async_t and add a regression test. PR-URL: nodejs#18051Fixes: nodejs#18047 Refs: nodejs#18020 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

jasnell · 2018-01-09T18:22:28Z

New CI now that #18051 landed: https://ci.nodejs.org/job/node-test-pull-request/12469/

Scheduling a PerformanceGCCallback should not keep the loop alive but due to the recent switch to using the native SetImmediate method, it does. Go back to using uv_async_t and add a regression test. Backport-PR-URL: #18050 PR-URL: #18051Fixes: #18047 Refs: #18020 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

jasnell · 2018-01-09T20:32:31Z

This should be ready to go

Add an aliased buffer for session and stream statistics, add a few more metrics PR-URL: #18020 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

jasnell · 2018-01-09T20:35:11Z

Landed in a02fcd2

Add an aliased buffer for session and stream statistics, add a few more metrics PR-URL: #18020 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

Scheduling a PerformanceGCCallback should not keep the loop alive but due to the recent switch to using the native SetImmediate method, it does. Go back to using uv_async_t and add a regression test. Backport-PR-URL: #18050 PR-URL: #18051Fixes: #18047 Refs: #18020 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Add an aliased buffer for session and stream statistics, add a few more metrics PR-URL: #18020 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

Add an aliased buffer for session and stream statistics, add a few more metrics PR-URL: nodejs#18020 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

Add an aliased buffer for session and stream statistics, add a few more metrics Backport-PR-URL: #20456 PR-URL: #18020 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

Scheduling a PerformanceGCCallback should not keep the loop alive but due to the recent switch to using the native SetImmediate method, it does. Go back to using uv_async_t and add a regression test. PR-URL: #18051Fixes: #18047 Refs: #18020 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Jan 6, 2018

addaleax approved these changes Jan 6, 2018
View reviewed changes

addaleax added http2 Issues or PRs related to the http2 subsystem. perf_hooks Issues and PRs related to the implementation of the Performance Timing API. labels Jan 8, 2018

addaleax mentioned this pull request Jan 8, 2018
http2: refactor read mechanism #18030
Closed
3 tasks

TimothyGu approved these changes Jan 8, 2018
View reviewed changes

Comment threadlib/internal/http2/util.js Outdated
TimothyGuJan 8, 2018
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could probably be merged with the line before.

mcollina reviewed Jan 8, 2018
View reviewed changes

http2: use aliased buffer for perf stats, add stats
e5ddf5b
Add an aliased buffer for session and stream statistics, add a few more metrics

jasnell force-pushed the http2-improve-perf branch from c416358 to e5ddf5bCompare January 8, 2018 22:05

jasnell force-pushed the http2-improve-perf branch from 1b0bdd8 to fbdf3b2Compare January 8, 2018 22:58

[Squash] Testing
89dfeb6

jasnell force-pushed the http2-improve-perf branch from fbdf3b2 to 89dfeb6Compare January 8, 2018 23:18

Fishrock123 mentioned this pull request Jan 8, 2018
test: investigate failing test-timers-unrefed-in-beforeexit #18047
Closed

apapirovski mentioned this pull request Jan 9, 2018
perf_hooks: fix scheduling regression #18051
Closed
3 tasks

jasnell closed this Jan 9, 2018

MylesBorins mentioned this pull request Jan 10, 2018
v9.4.0 proposal #18069
Merged

MylesBorins added the dont-land-on-v6.x label Jan 19, 2018

MylesBorins mentioned this pull request May 2, 2018
v8.11.2 proposal #20478
Merged

Uh oh!

Uh oh!

Conversation

jasnell commented Jan 6, 2018

Checklist

Affected core subsystem(s)

Uh oh!

addaleaxJan 6, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyGu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

apapirovski commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

apapirovski commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

apapirovski commented Jan 8, 2018

Uh oh!

addaleax commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

jasnell commented Jan 8, 2018

Uh oh!

addaleax commented Jan 8, 2018

Uh oh!

Fishrock123 commented Jan 8, 2018

Uh oh!

apapirovski commented Jan 8, 2018

Uh oh!

Fishrock123 commented Jan 8, 2018

Uh oh!

jasnell commented Jan 9, 2018

Uh oh!

jasnell commented Jan 9, 2018

Uh oh!

jasnell commented Jan 9, 2018

Uh oh!

jasnell commented Jan 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

addaleaxJan 6, 2018•
edited
Loading