Fatal Error (Abort) creating string larger than 268435440 bytes

[juliangruber]

Given this script:

var kMaxLength = process.binding('smalloc').kMaxLength;
console.log(kMaxLength, 'kMaxLength');

var len = 268435440;
var blob = new Buffer(len).toString('utf8');
console.log(len, 'ok');

len = 268435441;
blob = new Buffer(len).toString('utf8');
console.log(len, 'ok');

The output should be:

1073741823 'kMaxLength'
268435440 'ok'
268435441 'ok'

But it actually is:

1073741823 'kMaxLength'
268435440 'ok'


#
# Fatal error in ../deps/v8/src/handles.h, line 48
# CHECK(location_ != NULL) failed
#

==== C stack trace ===============================

 1: ??
 2: ??
 3: ??
 4: ??
 5: ??
 6: ??
 7: ??
 8: ??
 9: ??
Illegal instruction: 4

[juliangruber]

this is ~256mb

[petkaantonov]

@trevnorris @chrisdickinson and I are already working on this FWIW

[petkaantonov]

If your file is just using ascii, you can work around this currently by specifying 'ascii' as encoding

[Fishrock123]

I feel like this has already been reported, but I can't find the issue..

[juliangruber]

here is a stacktrace i get on linux:

==== C stack trace ===============================

 1: V8_Fatal
 2: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::String::NewStringType, int)
 3: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding)
 4: node::Buffer::Utf8Slice(v8::FunctionCallbackInfo<v8::Value> const&)
 5: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&))
 6: ??
 7: ??
Illegal instruction (core dumped)

[juliangruber]

updated the title, the issue is obviously the .toString()

[Fishrock123]

@juliangruber ah. Is this related to #649?

[petkaantonov]

@Fishrock123 yes, there is even a same issue reported deep in the comments.

It's not a v8 issue, while v8 does have a smallish limit for strings allocated in the js-heap (the 268mb or so), it is our fault for trying to allocate the string in js-heap in the first place.

[brendanashworth]

This doesn't seem to be an issue with the new Buffer rewrite / removal of smalloc:

brendanashworth in ~ $ node -v
v2.1.0
brendanashworth in ~ $ node test
1073741823 'kMaxLength'
268435440 'ok'
FATAL ERROR: invalid array length Allocation failed - process out of memory
Abort trap: 6

brendanashworth in ~ $ node -v
v3.0.0
brendanashworth in ~ $ node test
268435440 'ok'
268435441 'ok'

Closing this for now, feel free to reopen if I'm wrong! :)