Add support for zlib data format (RFC 1950)

[Ryder25]

Currently .Net doesn't support the zlib data format despite the majority of the work having been done by DeflateStream. I propose a new class be added similar to GZipStream to support RFC 1950.

Personally I'm encountering the zlib format in two areas. Many games depend on zlib, and save files using this format. We also use zlib in the firmware of one of our hardware products. We save debug data and compress it with zlib. For both of these scenarios, we use C# tooling, on the desktop, to operate on these files whether it is for the purpose of reading, modifying, or writing.

Due to the lack of support in .Net I either have to use lame hacky methods, or rely on a separate third party library for the functionality. This is such a shame when .Net already provides the majority of what's needed, and it should be trivial (as far I can tell) to include this as part of .Net.

While decompression is quite simple to do by skipping the 2-6 byte header, and 4 byte CRC at the end, I have trouble with compression. I wouldn't know what the proper header should be when using DeflateStream to do the compression, not to mention having to include a computation for the Adler32 checksum.

Here's a quick untested example of what I currently have to do for decompression to give you an idea of the "lame hacky-ness": (Note: I'm not checking the checksum here. I'm assuming the data is good.)

private byte[] ZlibDecompress(byte[] data)
{
 using (var ms = new MemoryStream())
 using (var ds = new DeflateStream(ms, CompressionMode.Decompress))
 using (var msOut = new MemoryStream())
 {
 int headerSize = (data[1] & 0b10000) == 0b10000 ? 6 : 2;
 ms.Write(data, headerSize, data.Length - 4 - headerSize);
 ms.Seek(0, SeekOrigin.Begin);
 ds.CopyTo(msOut);
 return msOut.ToArray();
 }
}

Proposed API

public class ZLibStream : Stream
{
 public ZLibStream(Stream stream, CompressionLevel compressionLevel);
 public ZLibStream(Stream stream, CompressionMode mode);
 public ZLibStream(Stream stream, CompressionLevel compressionLevel, bool leaveOpen);
 public ZLibStream(Stream stream, CompressionMode mode, bool leaveOpen);
 public override bool CanWrite { get; }
 public override bool CanSeek { get; }
 public override bool CanRead { get; }
 public Stream BaseStream { get; }
 public override long Length { get; }
 public override long Position { get; set; }
 public override void CopyTo(Stream destination, int bufferSize);
 public override Task CopyToAsync(Stream destination, int bufferSize, CancellationToken cancellationToken);
 public override ValueTask DisposeAsync();
 public override void Flush();
 public override Task FlushAsync(CancellationToken cancellationToken);
 public override int Read(Span<byte> buffer);
 public override int Read(byte[] array, int offset, int count);
 public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default);
 public override Task<int> ReadAsync(byte[] array, int offset, int count, CancellationToken cancellationToken);
 public override int ReadByte();
 public override long Seek(long offset, SeekOrigin origin);
 public override void SetLength(long value);
 public override void Write(byte[] array, int offset, int count);
 public override void Write(ReadOnlySpan<byte> buffer);
 public override Task WriteAsync(byte[] array, int offset, int count, CancellationToken cancellationToken);
 public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default);
 protected override void Dispose(bool disposing);
}

[JimBobSquarePants]

If adding ZLibStream it would be great to have access to the incremental 1-9 compression levels.

[kamronbatman]

I am forced to include a Zlib wrapper project and build a local nuget to keep my project as portable as possible.

We support Windows Server 2016/2019, Ubuntu 16/18/20LTS, CentOS 8.1, and Debian 9/10 against .NET Core 3.1.

Due to high concurrency and throughout requirements I can't use a fully managed solution or streams that have allocations. I am working on backend projects that utilize existing game client protocols, so It would be nice if I could reference the zlib native library directly, or a native library nuget is exposed (like libuv).

Lastly, I don't have access to modify the client code so I can't include headers/footers, nor would I want to.

I was looking forward to zlib implementation since ~2007 for multiple projects, but it looks like it is still not usable for my use cases.

[danmoseley]

@stephentoub unless this is committed for the 5.0 release, we should mark Future. It can still happen for 5.0, but milestone 5.0 == "debt to pay off" to successfully release.

[stephentoub]

We have a bug #38022. That needs to be fixed in some way for .NET 5. Implementing this is the right way to fix it. If we choose to punt on this, we can do the short term workaround there (which isn't a fix so much as disabling the functionality), but that's why I marked this as 5.0, to make a decision on the right thing to do. From my perspective, this whole thing can be done in a day.

[danmoseley]

Oh that's a good reason for the milestone then. 😸

[stephentoub]

For reference, other than tests, this is what the solution looks like:
stephentoub@8f73f68

[JimBobSquarePants]

@kamronbatman I've been developing a managed solution that is much faster than any other managed implementation (sometimes faster than DeflateStream) and allocates the same as DeflateStream. Dunno if it's of any interest as the planned implementation here does not expose many of the useful zlib options. (levels + mode)

I'm not finished cleaning up yet but will release something soonish.
https://github.com/SixLabors/ZlibStream