stream: support readable/writableHighWaterMark for Duplex streams by guymguym · Pull Request #14636 · nodejs/node

guymguym · 2017-08-04T23:34:05Z

Fixes: #14555

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

stream

gibfahn · 2017-08-05T09:52:48Z

cc/ @nodejs/streams

mcollina

Good work! There are only some nits to address!

mcollina · 2017-08-05T11:12:14Z

the copyright header is not needed for new files

mcollina · 2017-08-05T11:12:38Z

Can you remove this console.log?

mcollina · 2017-08-05T11:14:00Z

we use camelCase in general. You can use shorter names if you want.

guymguym · 2017-08-05T21:58:42Z

@mcollina Thanks for the review! I fixed the comments and squashed.

lpinca · 2017-08-06T08:29:31Z

CI: https://ci.nodejs.org/job/node-test-pull-request/9502/

guymguym · 2017-08-06T11:45:51Z

@lpinca I'm trying to understand if the failure is related to my change, and I suspect it doesn't.
Let me know if I'm missing anything.

lpinca · 2017-08-06T13:33:20Z

@guymguym you are correct, it's not related.

cjihrig

LGTM with a few comments.

cjihrig · 2017-08-07T15:32:51Z

Can you grab these 3 things using destructuring?

cjihrig · 2017-08-07T15:38:43Z

You could reduce duplication by doing something like this throughout the test:
[undefined,null,NaN].forEach((highWaterMark)=>{testTransform(666,DEFAULT,{ highWaterMark,readableHighWaterMark: 666});});

Fixes: nodejs#14555

guymguym · 2017-08-07T18:42:11Z

@cjihrig Thanks, fixed and squashed.

mcollina

LGTM

mcollina · 2017-08-08T09:35:28Z

Landed as c3c045a.

mcollina · 2017-08-08T09:35:42Z

Congratulations for your first contribution to Node.js!

mcollina · 2017-08-08T09:36:14Z

I've added the semver-minor flag as it's a new feature.

This commits adds support for readableHighWaterMark and writableHighWaterMark in Duplex stream, so that they can be set without accessing the internal state. Fixes: #14555 PR-URL: #14636 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

refack

A few comments a bit late. If we get some agreement I'll open a PR.

refack · 2017-08-08T17:01:27Z

 * `writableObjectMode` {boolean} Defaults to `false`. Sets `objectMode`
 for writable side of the stream. Has no effect if `objectMode`
 is `true`.
+ * `readableHighWaterMark` {number} Sets `highWaterMark` for the readable side


Suggestions:
Add a link to [#buffering]
Also in the first * (options) add links to [#constructor-new-streamwritableoptions] and [#new-streamreadableoptions]

refack · 2017-08-08T17:19:04Z

 for writable side of the stream. Has no effect if `objectMode`
 is `true`.
+ * `readableHighWaterMark` {number} Sets `highWaterMark` for the readable side
+ of the stream. Has no effect if `highWaterMark` is provided.


I'd phrase it as The general `highWaterMark` option overrides this one. IMHO a positive-active sentence is a little bit clearer than a negative-passive one.

I agree that positive is better. I just copied this format from readable/writableObjectMode.

refack · 2017-08-08T19:42:54Z

+ // However, some cases require setting options to different
+ // values for the readable and the writable sides of the duplex stream.
+ // These options can be provided separately as readableXXX and writableXXX.
+ var isDuplex = stream instanceof Stream.Duplex;


@mcollina isn't instanceof discouraged?

yes, but this was already there, this PR just moves it.

In my opinion it is best to allow it also for Readable/Writable without checking for Duplex, because this is an inherited behavior that behaves different on the base class vs the child class, and this is more confusing than helping. Not sure if this will be agreed by any developer reading the docs though. I wonder what you guys think.

refack · 2017-08-08T19:46:27Z

+});
+
+// test undefined, null, NaN
+[undefined, null, NaN].forEach((v) => {


what happens with ['', 'foo', ()=>{}, Symbol('frog'), {}, []]?

@refack The behavior will be unspecified in these cases, since highWaterMark is expected to be a number, but only checked to be a truthy value or 0.
This was also the case before my PR, though I would prefer it to be a specified behavior instead.
Perhaps the behavior should be to ignore non number values or else throw TypeError as a breaking change...

mcollina · 2017-08-08T20:39:01Z

@refack go ahead and fire a PR.

This commits adds support for readableHighWaterMark and writableHighWaterMark in Duplex stream, so that they can be set without accessing the internal state. Fixes: #14555 PR-URL: #14636 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

Ref: nodejs#14636

Ref: #14636 PR-URL: #14810 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

Notable changes * **Inspector** * `require()` is available in the inspector console now. [#8837](#8837) * **N-API** * New APIs for creating number values have been introduced. [#14573](#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](#14558) PR-URL: #14811

Notable changes * **HTTP2** * Experimental support for the built-in `http2` has been added via the `--expose-http2` flag. [#14239](#14239) * **Inspector** * `require()` is available in the inspector console now. [#8837](#8837) * Multiple contexts, as created by the `vm` module, are supported now. [#14465](#14465) * **N-API** * New APIs for creating number values have been introduced. [#14573](#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](#14558) PR-URL: #14811

Notable changes * **HTTP2** * Experimental support for the built-in `http2` has been added via the `--expose-http2` flag. [#14239](nodejs/node#14239) * **Inspector** * `require()` is available in the inspector console now. [#8837](nodejs/node#8837) * Multiple contexts, as created by the `vm` module, are supported now. [#14465](nodejs/node#14465) * **N-API** * New APIs for creating number values have been introduced. [#14573](nodejs/node#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](nodejs/node#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](nodejs/node#14558) PR-URL: nodejs/node#14811

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Aug 4, 2017

mcollina reviewed Aug 5, 2017
View reviewed changes

guymguym force-pushed the stream-issue-14555 branch from 771f20e to 078b4eeCompare August 5, 2017 21:56

cjihrig approved these changes Aug 7, 2017
View reviewed changes

stream: support readable/writableHighWaterMark for Duplex streams
13a8f7b
Fixes: nodejs#14555

guymguym force-pushed the stream-issue-14555 branch from 078b4ee to 13a8f7bCompare August 7, 2017 18:40

mcollina approved these changes Aug 8, 2017
View reviewed changes

mcollina closed this Aug 8, 2017

mcollina added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 8, 2017

refack reviewed Aug 8, 2017
View reviewed changes

addaleax added a commit to addaleax/node that referenced this pull request Aug 13, 2017
doc: add missing changes: metadata for streams
04c0387
Ref: nodejs#14636

addaleax mentioned this pull request Aug 13, 2017
doc: add missing changes: metadata for recent PRs #14810
Closed

addaleax added a commit that referenced this pull request Aug 13, 2017
doc: add missing changes: metadata for streams
7e0c2ea
Ref: #14636 PR-URL: #14810 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

addaleax added a commit that referenced this pull request Aug 13, 2017
doc: add missing changes: metadata for streams
4811fea
Ref: #14636 PR-URL: #14810 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

addaleax mentioned this pull request Aug 13, 2017
v8.4.0 proposal #14811
Merged

MylesBorins mentioned this pull request Jan 15, 2018
Semver Minor Tracking nodejs/Release#298
Closed

everhardt mentioned this pull request Dec 9, 2019
[node] stream: support readable/writableHWM for Duplex DefinitelyTyped/DefinitelyTyped#40843
Merged
9 tasks

Uh oh!

Uh oh!

Conversation

guymguym commented Aug 4, 2017

Checklist

Affected core subsystem(s)

Uh oh!

gibfahn commented Aug 5, 2017

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guymguym commented Aug 5, 2017

Uh oh!

lpinca commented Aug 6, 2017

Uh oh!

guymguym commented Aug 6, 2017

Uh oh!

lpinca commented Aug 6, 2017

Uh oh!

cjihrig left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guymguym commented Aug 7, 2017

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

mcollina commented Aug 8, 2017

Uh oh!

mcollina commented Aug 8, 2017

Uh oh!

mcollina commented Aug 8, 2017

Uh oh!

refack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcollina commented Aug 8, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone