test: add tests for invalid UTF-8

[git-srinivas]

Fixes: #39804

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• make lint passes

[lpinca]

// Run with --expose-internals flagconstassert=require('assert');const{ toUSVString }=require('internal/util');constdecoder=newTextDecoder();constchunk=Buffer.from([0x66,0x6f,0x6f,0xed,0xa0,0x80]);// foo + U+D800conststr=decoder.decode(chunk);assert.strictEqual(toUSVString(str),'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

[lpinca]

Hmm both Chrome and Safari work like this PR:

newBlob([newUint8Array([0x66,0x6f,0x6f,0xed,0xa0,0x80])]).text().then((str)=>str==='foo\ufffd\ufffd\ufffd').then(console.log);

but this is not consistent with

const{ toUSVString }=require('internal/util');toUSVString('foo\ud800');// returns 'foo\ufffd';

[git-srinivas]

// Run with --expose-internals flagconstassert=require('assert');const{ toUSVString }=require('internal/util');constdecoder=newTextDecoder();constchunk=Buffer.from([0x66,0x6f,0x6f,0xed,0xa0,0x80]);// foo + U+D800conststr=decoder.decode(chunk);assert.strictEqual(toUSVString(str),'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

@lpinca Does it mean the issue 39804 is not valid?

[ronag]

```js
// Run with --expose-internals flag
const assert = require('assert');
const { toUSVString } = require('internal/util');
const decoder = new TextDecoder();
const chunk = Buffer.from([0x66, 0x6f, 0x6f, 0xed, 0xa0, 0x80]); // foo + U+D800
const str = decoder.decode(chunk);
assert.strictEqual(toUSVString(str), 'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

@lpinca Does it mean the issue 39804 is not valid?

Does your tests pass without the fix?

[git-srinivas]

yes. I ran test cases with --expose-internals flag. All of my test cases are passing without the fix.

[ronag]

Then I don't think this needs fixing. I would propose you remove the changes but keep the tests. We should still merge the tests.

[git-srinivas]

// Run with --expose-internals flagconstassert=require('assert');const{ toUSVString }=require('internal/util');constdecoder=newTextDecoder();constchunk=Buffer.from([0x66,0x6f,0x6f,0xed,0xa0,0x80]);// foo + U+D800conststr=decoder.decode(chunk);assert.strictEqual(toUSVString(str),'foo\ufffd');

TextDecoder() already replaces each byte of the surrogate code point with U+FFFD.

@lpinca@ronag
I am trying to understand this example to write some good testcases.
May i know how surrogate U+D800 is encoded as [0xed, 0xa0, 0x80]?

I tried with Buffer.from('\ud800','utf8') and what i get is <Buffer ef bf bd> .

[lpinca]

All surrogates are 3 bytes and in this range:

byte 1 = ED
byte 2 = [A0 - BF]
byte 3 = [80 - BF]

I tried with Buffer.from('\ud800','utf8') and what i get is <Buffer ef bf bd> .

That's because Buffer.from() replaces invalid UTF-8 byte sequences with U+FFFD.

[lpinca]

This does not make much sense because passthrough.write() and passthrough.end() will call Buffer.from() when the chunk is written. If anything, chunks should be Buffers or Uint8Arrays.

[lpinca]

Similar to the one above this does not make much sense to me. When decoder.decode(chunk) is called, chunk is already U+FFFD (valid UTF-8).

[lpinca]

@git-srinivas if you can please

1. Update PR title to update the actual scope of the PR
2. Squash commits and use a proper commit title and body as per point 1.

[git-srinivas]

Sure @lpinca I'll do the changes

[lpinca]

@git-srinivas can I suggest something like this?

test: add tests for invalid UTF-8
Verify that `Blob.prototype.text()`, `streamConsumers.text()` and
`TextDecoder.prototype.decode()` work as expected with invalid UTF-8.
Fixes: https://github.com/nodejs/node/issues/39804

I find the current commit message a bit misleading becausetoUSVString() works differently as per #40351 (comment).

Thank you.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40831/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40852/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40856/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40860/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40861/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40868/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40876/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40885/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40897/

[lpinca]

Landed in dc35aef.