zlib: Allow zlib.gunzip features with zlib.unzip

[addaleax]

Pull Request check-list

• Does make -j8 test (UNIX) or vcbuild test nosign (Windows) pass with
  this change (including linting)?
• Is the commit message formatted according to CONTRIBUTING.md?
• If this change fixes a bug (or a performance problem), is a regression
  test (or a benchmark) included?

Affected core subsystem(s)

zlib

Description of change

Detect whether a gzip file is being passed to unzip* by testing the first bytes for the gzip magic bytes, and setting the decompression mode to GUNZIP or INFLATE according to the result.

This enables gzip-only features like multi-member support to be used together with the unzip* autodetection support and thereby makes gunzip* and unzip* return identical results for gzip input again.

This is more or less a follow-up change to #5120.

[bnoordhuis]

LGTM. CI: https://ci.nodejs.org/job/node-test-pull-request/2051/

I asked you in the other PR to remove the defines but that was before I'd seen this PR.

[addaleax]

Sorry, even if CI passes, I’ll have to update this with a somewhat different approach and more tests.
On the JS side, currently the assumption is made that availInAfter > 0 implies availOutAfter === 0, but with the break; statement introduced here that would no longer be the case.

[addaleax]

Okay, added a new commit with tests for the case that even the magic bytes don’t come in a single chunk (I know it’s quite an unlikely edge case, but I guess even that should be supported). The new test will require the bugfix from #5883 to pass, though.

[kthelgason]

This looks like something that should obviously be supported. The only relevant bytes here are the first two bytes of the file, right? When would these be in separate chunks?

[addaleax]

Yes, these are the only inspected bytes. And honestly, I have no idea how they might get split into separate chunks. 😄 But it could, because we don’t have control about where the data comes from, and I think that means either not supporting this feature at all or creating an edge case in which it does not work.
And if you have a better idea on how to implement this, I’m all for it! I’m pretty sure there must be more beautiful solutions than the nested switch in node_zlib.cc.

[kthelgason]

I guess if someone set the chunksize to 1 when constructing a Gunzip object it could happen, whether that's a very clever thing to do is another matter! In any case I agree that we should try to support every case that can occur or none at all.

[addaleax]

>zlib.createGunzip({chunkSize:1})
Error: Invalid chunksize: 1

I guess you’re talking about something like receiving Z_STREAM_END from zlib exactly when the input buffer ends with a gzip member? I don’t think that’s actually a problem, Z_STREAM_END gets pretty much treated the same as Z_OK if I read the code correctly.

[kthelgason]

Actually, your example is exactly what I was thinking of, but I'm happy to see that it isn't possible. But obviously one can construct pathological cases where this would happen, like passing one byte at a time to unzip.write.

Anyway, I give this a big 👍, thanks for improving on this 😃

[addaleax]

I think that should be covered by the test cases here, but I guess you can feel free to create more of them.

And again, good to have you on board with this! 😄

[bnoordhuis]

Quick question: would it be necessary to check that ctx->strm_.avail_in > 0 before trying to read the first byte?

[addaleax]

You’re right.

[bnoordhuis]

In that vein, shouldn't there be a similar check for GZIP_HEADER_ID2?

Another question: what happens when the state machines sees GZIP_HEADER_ID1 followed by a byte that's not GZIP_HEADER_ID2? Is the first byte lost?

[addaleax]

Sorry, you’re right – updated again. And no, no bytes are lost in that case; there is always the fall-through in the outer switch from case UNZIP: to case INFLATE:, so all input ends up being passed to inflate().

[addaleax]

Rebased and updated with your suggestion; The tests are passing now that #5883 is merged.

[jasnell]

LGTM if @bnoordhuis is happy.

[jasnell]

CI: https://ci.nodejs.org/job/node-test-pull-request/2121/

[bnoordhuis]

LGTM with the style nit that comments should be capitalized and punctuated.

[addaleax]

Updated the comments. :)

[bnoordhuis]

Thanks Anna, landed in 2d7e316.