crypto: make malloc failure check cross-platform

[Trott]

Checklist

• make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
• commit message follows commit guidelines

Affected core subsystem(s)

crypto

Description of change

malloc(0) may return NULL on some platforms. Do not report
out-of-memory error unless malloc was passed a number greater than
0.

Marking as in progress for the moment. This is an attempt to fix some of the perma-flaky tests on AIX.

[Trott]

CI: https://ci.nodejs.org/job/node-test-pull-request/3900/

[Trott]

This seems to work. Removing in progress label. /cc @mhdawson @nodejs/crypto

Hopefully this is an acceptable short term workaround if it is not acceptable as a permanent fix.

Guess I better start looking around the code to see if anything bad could happen to that null pointer...

[Trott]

Fixes three of the four current perma-flaky tests on AIX. SmartOS failure looks unrelated.

CI again: https://ci.nodejs.org/job/node-test-pull-request/3901/

[bnoordhuis]

See also #7564. I wasn't very happy with that approach but, mea culpa, I didn't get around to posting the alternative solution I talked about. I suppose any fix is better than no fix at this point.

[indutny]

LGTM, if it works

[addaleax]

LGTM

[jasnell]

LGTM

[mhdawson]

I'd prefer to land #7564 as it addresses the general problem as opposed to the specific problem that this patch addresses. However, I'm not opposed to landing this as well as it would be better not to depend on the platform specific malloc(0) behavior in this specific case.

So LTGM to land this but @bnoordhuis can you also give me an LTGM on #7564 as well so that I could land it.

[Trott]

Would I be correct to infer that folks providing LGTM comments are affirming that they believe using the null pointer will have no detrimental effects on security or stability? (That's my opinion, but I don't trust myself to analyze C++ crypto code, so I'm looking for specific external validation on this.)

[addaleax]

@Trott Yes, using nullptr for a zero-length Buffer should be okay. We’re definitely having tests to make sure that that works. :)

[mhdawson]

Based on the way that @bnoordhuis suggested that I change the implementation we'll need this fix as it moves all platforms so that we'll consistently get a nullptr for malloc(0).

To be able to test I cherry picked this across/resolved merge issues so might make sense to land them together. Just about to push 2 commits to #7564 and then do a CI run to make sure all it ok across platforms.

[mhdawson]

Landed as ed640ae

[MylesBorins]

@mhdawson should this be backported?

[MylesBorins]

@Trott backport?

edit: does not land cleanly

[Trott]

@thealphanerd I believe it is only needed if and only if https://github.com/nodejs/node/commit/a00ccb0fb9eb716925058b0a20fcec9251de3309/ / #7564 lands. That PR is currently marked as "do not land" so I think this too should be marked that way.

[mhdawson]

Agreed.