buffer: fix single-character string filling by addaleax · Pull Request #9837 · nodejs/node

addaleax · 2016-11-29T11:12:21Z

Checklist

make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

buffer

Description of change

Fix the fast path for buffer.fill() with a single-character string.

The fast path only works for strings that are equivalent to a
single-byte buffer, but that condition was not checked properly
for the utf8 or utf16le encodings and is always true for the
latin1 encoding.

This change fixes these problems.

Fixes: #9836

/cc @nodejs/buffer
~~CI: https://ci.nodejs.org/job/node-test-commit/6222/~~
~~CI: https://ci.nodejs.org/job/node-test-commit/6224/~~
CI: https://ci.nodejs.org/job/node-test-commit/6600/

bnoordhuis · 2016-11-29T11:26:32Z

This is going to throw Unknown encoding: undefined.

@bnoordhuis Right, updated!

bnoordhuis · 2016-11-29T11:46:45Z

Test failures look related. I would guess it's an endianness issue.

not ok 27 parallel/test-buffer-fill
---
duration_ms: 0.139
severity: fail
stack: |-
assert.js:85
throw new assert.AssertionError({
^
AssertionError: '愀愀愀愀愀愀愀愀' === 'aaaaaaaa'
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcbe-ubuntu1404/test/parallel/test-buffer-fill.js:394:8)

addaleax · 2016-11-29T13:18:32Z

I’ve updated the test file here to see whether the failure is related to my changes in lib/buffer.js or not, and it looks like this just exposed the problem:

assert.deepStrictEqual(Buffer.allocUnsafeSlow(16).fill('ab','utf16le'),Buffer.from('61006200610062006100620061006200','hex'));

results in

AssertionError: <Buffer00610062006100620061006200610062> deepStrictEqual <Buffer61006200610062006100620061006200>

on big endian machines (example ci run).

Not sure why that is (I think we’re already testing UCS2 on big-endian machines), I’ll take a look later unless somebody else wants to.

addaleax · 2016-12-06T02:46:11Z

Finally got back to this; Updated, new CI @ https://ci.nodejs.org/job/node-test-commit/6451/

Apparently the tests expected a swapped byte order of #fill on big-endian platforms for utf16le, so I needed to fix that up, too. (I think it’s safe to say that the expected behaviour for an encoding called utf16le is to always have little-endian output.)

@nodejs/buffer PTAL

trevnorris

Looks good. I'll sign off once the final test is fixed.

trevnorris · 2016-12-06T05:26:12Z

For backwards compatibility do we still need to check for 'binary', or is that no longer supported? If this is back ported then it'll probably need to be checked there.

@trevnorris I switched the blocks around here so that this is after normalizeEncoding, so there’s no need to worry about alternate names

cool. thanks.

trevnorris · 2016-12-06T05:26:35Z

From your comment sounds like just this needs to be removed right?

Sorry, I should have made this clearer, but this is already the working version; i.e. this line was missing and should already have been there afaict

addaleax

New CI since the last one had build failures: https://ci.nodejs.org/job/node-test-commit/6469/

(everything should be green here in theory)

addaleax · 2016-12-06T12:35:10Z

@trevnorris I switched the blocks around here so that this is after normalizeEncoding, so there’s no need to worry about alternate names

addaleax · 2016-12-06T12:36:12Z

Sorry, I should have made this clearer, but this is already the working version; i.e. this line was missing and should already have been there afaict

jasnell

LGTM once @trevnorris is happy with it

trevnorris · 2016-12-09T22:31:57Z

LGTM

addaleax · 2016-12-11T22:36:32Z

@bnoordhuis Any further comments here? Otherwise I’d like to land this within the next few days.

bnoordhuis

LGTM with a suggestion.

bnoordhuis · 2016-12-11T23:33:08Z

Suggestion: maybe use String#repeat() for the other strings as well, it looks more Obviously Correct.

@bnoordhuis done, makes sense

There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour.

Fix the fast path for `buffer.fill()` with a single-character string. The fast path only works for strings that are equivalent to a single-byte buffer, but that condition was not checked properly for the `utf8` or `utf16le` encodings and is always true for the `latin1` encoding. This change fixes these problems. Fixes: nodejs#9836

addaleax · 2016-12-12T14:34:20Z

New CI: https://ci.nodejs.org/job/node-test-commit/6600/

addaleax · 2016-12-12T15:59:12Z

Landed in 9ee915b and d4f00fe

There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour. PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Fix the fast path for `buffer.fill()` with a single-character string. The fast path only works for strings that are equivalent to a single-byte buffer, but that condition was not checked properly for the `utf8` or `utf16le` encodings and is always true for the `latin1` encoding. This change fixes these problems. Fixes: #9836 PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour. PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Fix the fast path for `buffer.fill()` with a single-character string. The fast path only works for strings that are equivalent to a single-byte buffer, but that condition was not checked properly for the `utf8` or `utf16le` encodings and is always true for the `latin1` encoding. This change fixes these problems. Fixes: #9836 PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Notable changes: - buffer: - fix single-character string filling (Anna Henningsen) #9837 - handle UCS2.fill() properly on BE (Anna Henningsen) #9837 - url: - including base argument in originFor (joyeecheung) #10021 - improve URLSearchParams spec compliance (Timothy Gu) #9484 - http: - remove stale timeout listeners (Karl Böhlmark) #9440 - build: - fix node_g target (Daniel Bevenius) #10153 PR-URL: #10277

Notable changes SEMVER-MINOR - url: - add inspect function to TupleOrigin (Safia Abdalla) #10039 - crypto: - allow adding extra certs to well-known CAs (Sam Roberts) #9139 SEMVER-PATCH - buffer: - fix single-character string filling (Anna Henningsen) #9837 - handle UCS2 .fill() properly on BE (Anna Henningsen) #9837 - url: - including base argument in originFor (joyeecheung) #10021 - improve URLSearchParams spec compliance (Timothy Gu) #9484 - http: - remove stale timeout listeners (Karl Böhlmark) #9440 - build: - fix node_g target (Daniel Bevenius) #10153 - fs: - remove unused argument from copyObject() (Ethan Arrowood) #10041 - timers: - fix handling of cleared immediates (hveldstra) #9759 PR-URL: #10277

Notable changes: * **crypto**: - Allow adding extra certificates to well-known CAs. (Sam Roberts) [#9139](#9139) * **buffer**: - Fix single-character string filling. (Anna Henningsen) [#9837](#9837) - Handle UCS2 `.fill()` properly on BE. (Anna Henningsen) [#9837](#9837) * **url**: - Add inspect function to TupleOrigin. (Safia Abdalla) [#10039](#10039) - Including base argument in originFor. (joyeecheung) [#10021](#10021) - Improve URLSearchParams spec compliance. (Timothy Gu) [#9484](#9484) * **http**: - Remove stale timeout listeners. (Karl Böhlmark) [#9440](#9440) * **build**: - Fix node_g target. (Daniel Bevenius) [#10153](#10153) * **fs**: - Remove unused argument from copyObject(). (EthanArrowood) [#10041](#10041) * **timers**: - Fix handling of cleared immediates. (hveldstra) [#9759](#9759) * **src**: - Add wrapper for process.emitWarning(). (SamRoberts) [#9139](#9139) - Fix string format mistake for 32 bit node.(Alex Newman) [#10082](#10082)

Notable changes: * **crypto**: The built-in list of Well-Known CAs (Certificate Authorities) can now be extended via a NODE_EXTRA_CA_CERTS environment variable. (Sam Roberts) [#9139](#9139) * **buffer**: buffer.fill() now works properly for the UCS2 encoding on Big-Endian machines. (Anna Henningsen) [#9837](#9837) * **url**: - Including base argument in URL.originFor() to meet specification compliance. (joyeecheung) [#10021](#10021) - Improve URLSearchParams to meet specification compliance. (Timothy Gu) [#9484](#9484) * **http**: Remove stale timeout listeners in order to prevent a memory leak when using keep alive. (Karl Böhlmark) [#9440](#9440)

Notable changes: * buffer: - buffer.fill() now works properly for the UCS2 encoding on Big-Endian machines. (Anna Henningsen) nodejs#9837 * cluster: - disconnect() now returns a reference to the disconnected worker. (Sean Villars) nodejs#10019 * crypto: - The built-in list of Well-Known CAs (Certificate Authorities) can now be extended via a NODE_EXTRA_CA_CERTS environment variable. (Sam Roberts) nodejs#9139 * http: - Remove stale timeout listeners in order to prevent a memory leak when using keep alive. (Karl Böhlmark) nodejs#9440 * tls: - Allow obvious key/passphrase combinations. (Sam Roberts) nodejs#10294 * url: - Including base argument in URL.originFor() to meet specification compliance. (joyeecheung) nodejs#10021 - Improve URLSearchParams to meet specification compliance. (Timothy Gu) nodejs#9484 PR-URL: nodejs#10277

Notable changes: * buffer: - buffer.fill() now works properly for the UCS2 encoding on Big-Endian machines. (Anna Henningsen) #9837 * cluster: - disconnect() now returns a reference to the disconnected worker. (Sean Villars) #10019 * crypto: - The built-in list of Well-Known CAs (Certificate Authorities) can now be extended via a NODE_EXTRA_CA_CERTS environment variable. (Sam Roberts) #9139 * http: - Remove stale timeout listeners in order to prevent a memory leak when using keep alive. (Karl Böhlmark) #9440 * tls: - Allow obvious key/passphrase combinations. (Sam Roberts) #10294 * url: - Including base argument in URL.originFor() to meet specification compliance. (joyeecheung) #10021 - Improve URLSearchParams to meet specification compliance. (Timothy Gu) #9484 PR-URL: #10277

Notable changes: * buffer: - buffer.fill() now works properly for the UCS2 encoding on Big-Endian machines. (Anna Henningsen) nodejs/node#9837 * cluster: - disconnect() now returns a reference to the disconnected worker. (Sean Villars) nodejs/node#10019 * crypto: - The built-in list of Well-Known CAs (Certificate Authorities) can now be extended via a NODE_EXTRA_CA_CERTS environment variable. (Sam Roberts) nodejs/node#9139 * http: - Remove stale timeout listeners in order to prevent a memory leak when using keep alive. (Karl Bohlmark) nodejs/node#9440 * tls: - Allow obvious key/passphrase combinations. (Sam Roberts) nodejs/node#10294 * url: - Including base argument in URL.originFor() to meet specification compliance. (joyeecheung) nodejs/node#10021 - Improve URLSearchParams to meet specification compliance. (Timothy Gu) nodejs/node#9484 PR-URL: nodejs/node#10277 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

MylesBorins · 2017-01-23T02:10:32Z

@addaleax this lands cleany on v6.x but no v4.x. Could you backport?

There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour. PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Fix the fast path for `buffer.fill()` with a single-character string. The fast path only works for strings that are equivalent to a single-byte buffer, but that condition was not checked properly for the `utf8` or `utf16le` encodings and is always true for the `latin1` encoding. This change fixes these problems. Fixes: #9836 PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour. PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Fix the fast path for `buffer.fill()` with a single-character string. The fast path only works for strings that are equivalent to a single-byte buffer, but that condition was not checked properly for the `utf8` or `utf16le` encodings and is always true for the `latin1` encoding. This change fixes these problems. Fixes: #9836 PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

addaleax · 2017-01-26T21:52:37Z

I don’t think this needs to be backported, the v4.x equivalent of the tests here are passing.

There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour. PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Fix the fast path for `buffer.fill()` with a single-character string. The fast path only works for strings that are equivalent to a single-byte buffer, but that condition was not checked properly for the `utf8` or `utf16le` encodings and is always true for the `latin1` encoding. This change fixes these problems. Fixes: #9836 PR-URL: #9837 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

nodejs-github-bot added the buffer Issues and PRs related to the buffer subsystem. label Nov 29, 2016

addaleax added lts-watch-v4.x labels Nov 29, 2016

bnoordhuis requested changes Nov 29, 2016
View reviewed changes

addaleax force-pushed the buffer-fix-single-byte-fill branch from 0690d5f to 641cfa2Compare November 29, 2016 11:31

addaleax force-pushed the buffer-fix-single-byte-fill branch 5 times, most recently from c7c27c9 to 63e0a5eCompare November 29, 2016 13:14

addaleax force-pushed the buffer-fix-single-byte-fill branch 3 times, most recently from fb755ba to 6790a6fCompare December 6, 2016 02:28

trevnorris reviewed Dec 6, 2016
View reviewed changes

addaleax commented Dec 6, 2016
View reviewed changes

jasnell approved these changes Dec 6, 2016
View reviewed changes

bnoordhuis approved these changes Dec 11, 2016
View reviewed changes

addaleax added 2 commits December 12, 2016 15:33

buffer: handle UCS2 .fill() properly on BE
cea2194
There was a byte-order mismatch for `buffer#fill` on big-endian platforms. Weirdly, the tests seemed to expect that wrong behaviour.

addaleax force-pushed the buffer-fix-single-byte-fill branch from 6790a6f to b0a80b7Compare December 12, 2016 14:33

addaleax closed this Dec 12, 2016

addaleax deleted the buffer-fix-single-byte-fill branch December 12, 2016 15:59

italoacasas mentioned this pull request Dec 15, 2016
v7.3.0 proposal #10277
Merged

MylesBorins added land-on-v6.x and removed lts-watch-v6.x labels Jan 23, 2017

MylesBorins mentioned this pull request Jan 24, 2017
v6.10.0 proposal #10974
Merged

addaleax added dont-land-on-v4.x and removed lts-watch-v4.x labels Jan 26, 2017

Uh oh!

Uh oh!

Conversation

addaleax commented Nov 29, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Affected core subsystem(s)

Description of change

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnoordhuis commented Nov 29, 2016

Uh oh!

addaleax commented Nov 29, 2016

Uh oh!

addaleax commented Dec 6, 2016

Uh oh!

trevnorris left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

addaleax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasnell left a comment

Choose a reason for hiding this comment

Uh oh!

trevnorris commented Dec 9, 2016

Uh oh!

addaleax commented Dec 11, 2016

Uh oh!

bnoordhuis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

addaleax commented Dec 12, 2016

Uh oh!

addaleax commented Dec 12, 2016

Uh oh!

MylesBorins commented Jan 23, 2017

Uh oh!

addaleax commented Jan 26, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

addaleax commented Nov 29, 2016•
edited
Loading