stream: Support writing null to stream.Transform

[kevinoid]

Here's a proposed fix for #5711, which passes null to ._transform, like other falsey values.

It updates the falsey value test for Transform to include null (and undefined, which was previously omitted) and adds a test for the falsey value behavior of Writable in objectMode, as requested in #5711 (comment)

Let me know what you think @mcollina and @calvinmetcalf (and anyone else interested, of course).

Fixes: #5711

[mscdex]

Does this mean that non-objectMode streams can receive non-Buffer/string values now or do they get stringified or what?

[kevinoid]

The changes in this PR don't affect non-objectMode streams, since the writes don't make it to ._write().

Currently calling .write(null) causes TypeError: Cannot read property 'length' of null at _stream_writable.js:268. It would probably make more sense to catch it in validChunk, but that change might be better suited for another PR?

[kevinoid]

Looking closer, .write(null) for non-objectMode streams increments pendingcb before throwing, so the stream won't ever 'finish'. Looks like a good thing to fix. Should I include it in this PR or roll another for the non-objectMode behavior?

[mcollina]

Roll another PR.

We should probably run this on citgm (https://github.com/nodejs/citgm) before landing, just in case.

[kevinoid]

@mcollina Sure, can't hurt. Is there a build farm which runs citgm? Is there anything I need to do to schedule a run?

[mcollina]

Not sure. Any idea @mafintosh@calvinmetcalf?

[jasnell]

@thealphanerd set up a job in ci.nodejs.org that runs citgm. I'm sure he can help get you going with it.

[calvinmetcalf]

what was the original rational for this check?

[kevinoid]

26ca7d7 describes it as "null is reserved to indicate stream eof". Perhaps the author confused .write(null) with .push(null), or planned for .write(null) to cause EOF? Or they intended to disallow .write(null) to avoid confusion about it being interpreted as EOF in other contexts? I'm not exactly sure.

[mafintosh]

this change also means that writing null to a passthrough will end the readable side which is a bit weird

[kevinoid]

this change also means that writing null to a passthrough will end the readable side

I don't think so. null and undefined are ignored by the ._transform callback.

[mafintosh]

ah cool. didn't know that.

[MylesBorins]

citgm: https://ci.nodejs.org/job/thealphanerd-smoker/162/

[kevinoid]

Thanks @thealphanerd!

[mcollina]

Quite a few failures, we should probably have a closer look at them.

[mcollina]

Thinking about this, there is a good reason why you cannot pass null to through a Transform stream.
It is the PassThrough scenario.

stream.write(null) -> stream._transform(null, '', cb) -> this.push(null) -> the stream closes

For simmetry, we might even remove writing null to a Writable.

However, we should support the other falsey values (like undefined).

[kevinoid]

Indeed. stream.PassThrough works because it passes the written value to the _transform callback, which ignores null and undefined, but it looks like there are plenty of other implementations which pass the written value to push without filtering null.

I can open a new PR which ignores, throws, or emits an error for .write(null). Is there agreement on which of these should occur? I see the following options:

• Match the current behavior for non-objectMode streams by throwing TypeError, as mentioned in stream: Support writing null to stream.Transform #5729 (comment) . This seems like unintentional behavior which is likely to cause uncaughtException, but it would be consistent.
• Behave like non-string/non-Buffer values written to non-objectMode streams, and emit an error from validChunk. I'm not a big fan of this behavior, since reading code is likely to think the stream has ended with an error while the writing code is likely to keep writing.
• Call the write callback without calling _transform as @calvinmetcalf had suggested in stream: Writing null to objectMode Transform is unrecoverable #5711 (comment) . This seems like a bit of a stretch for "successfully writing" the value, but it also seems like the least likely to cause breakage.

My vote is for the last option. In an ideal world, I would like to change validChunk to only emit 'error' when the caller did not provide a callback, to avoid confusing readers listening for 'error', but this would be a backwards-incompatible change that is probably not justified. So my vote is to call the callback without an Error. What do you guys think? Should the behavior of non-objectMode streams be changed too?

[mcollina]

@kevinoid I'm not understanding all the various options you are laying out. Can you please clarify?

I think we should emit/callback an Error if null or undefined are passed in write() independently of objectMode. Regarding the fact that the stream might still be used after such an error, this is highly unlikely: an user will need to check the Error' type (or message) and recover from there. It is better to jettison all the stream and restart from scratch.

[kevinoid]

@mcollina Sure, that will work. I disagree that the writer continuing is unlikely, since I think there is a lot of code in the wild which doesn't properly check for write errors. But I'm not opposed to that option.

Stated somewhat differently (in the same order), the three options I see are:

1. Make write(null) in objectMode behave like write(null) in non-objectMode by throwing TypeError.
2. Make null fail the checks in validChunk, which will emit 'error' and pass Error to callback.
3. Ignore null writes without emitting or returning Error. Call cb without error on nextTick.

Should I open a PR to implement option 2, as you suggested, or hold off until after it is discussed at the WG meeting?

[mcollina]

I'm ok for option 2. Calvin?

Il giorno gio 31 mar 2016 alle 20:19 Kevin Locke notifications@github.com
ha scritto:

@mcollinahttps://github.com/mcollina Sure, that will work. I disagree
that the writer continuing is unlikely, since I think there is a lot of
code in the wild which doesn't properly check for write errors. But I'm not
opposed to that option.

Stated somewhat differently (in the same order), the three options I see
are:

1. Make write(null) in objectMode behave like write(null) in non-
   objectMode by throwing TypeError.
2. Make null fail the checks in validChunk, which will emit 'error'
   and pass Error to callback.
3. Ignore null writes without emitting or returning Error. Call cb
   without error on nextTick.

Should I open a PR to implement option 2, as you suggested, or hold off
until after it is discussed at the WG meeting?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#5729 (comment)

[calvinmetcalf]

at the readable stream meeting yesterday we decide on 2 so I'll open a pull for that

[kevinoid]

Sounds good, thanks @calvinmetcalf!