Large strings decoded from latin1 and then encoded to utf8 has wrong size

[henrik-ep]

• Version: v8.11.4
• Platform: Linux xyz 4.17.5-200.fc28.x86_64 deps: update openssl to 1.0.1j #1 SMP Tue Jul 10 13:39:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem:

Decoding a latin1 buffer larger than about 1MB to a string and encoding that string into utf-8 gives a buffer with the same number of bytes as the latin1 input even though more are required for characters that use more space in utf-8.

This seems to work properly on v10.x but not v8.x or v9.x.

Code that demonstrates the problem:

const fs = require('fs');
const s = 'Räksmörgås';
let ss = '';

const SIZE = (1024 * 1024);
// works:
//const SIZE = (1024 * 512);

while (ss.length < SIZE) {
 ss = ss + ss.length + ' ' + s + '\n';
}

// create latin1 buffer we can decode
let l1Buffer = Buffer.from(ss, 'latin1');
let l1String = l1Buffer.toString('latin1')
// also fixes it:
// l1String = ('x' + l1String).substring(1, l1String.length + 1);
// create utf8 buffer from decoded latin1 string
let u8Buffer = Buffer.from(l1String, 'utf8')

console.log(l1Buffer.length);
console.log(u8Buffer.length);

if (l1Buffer.length === u8Buffer.length) {
 console.log('failed, should be different size');
} else {
 console.log('works');
}

[jasnell]

Quick test on 8.11.4 confirms the difference in behavior between Node.js 8 and 10.

/cc @nodejs/intl @nodejs/buffer @srl295

[addaleax]

This was fixed by #18216, I think.

[jasnell]

Yep, makes sense. That PR is still pending a backport to 8.x. Once the backport happens, this issue should be resolved.

[addaleax]

→ Backport is in #22731

[starkwang]

This issue is fixed in #18216 and #22731 so I'd like to close it.