std.compress.flate: rework decompression and delete compression

[andrewrk]

Followup from #24329

There were a lot of problems with this code, so I opted to reduce scope.

The compression code I don't think is worth saving. It will need to be recontributed, with a lot more scrutiny.

Next up: complete rewrite of std.http both Client and Server

Upgrade Guide

Deleted APIs:

• delete std.Io.SeekableStream
  • Instead, use *std.fs.File.Reader, *std.fs.File.Writer, or std.ArrayListUnmanaged concrete types, because the implementations will be fundamentally different based on whether you are operating on files or memory.
• delete std.Io.BitReader
  • Bit reading should not be astracted at this layer; it just makes your hot loop harder to optimize. Tightly couple this code with your stream implementation.
• delete std.Io.BitWriter
  • ditto
• std.compress API restructured everything to do with flate, which includes zlib and gzip
  • std.compress.flate.Decompress is your main API now and it has a container parameter

New API example:

vardecompress_buffer: [std.compress.flate.max_window_len]u8=undefined;
vardecompress: std.compress.flate.Decompress= .init(reader, .zlib, &decompress_buffer);
constdecompress_reader: *std.Io.Reader=&decompress.reader;

If decompress_reader will be piped entirely to a particular *Writer, then give it an empty buffer:

vardecompress: std.compress.flate.Decompress= .init(reader, .zlib, &.{});
constn=trydecompress.streamRemaining(writer);

Compression

Sorry, you will have to copy the old code into your application, or use a third party package.

It will be nice to get deflate back into the Zig standard library, but for now, progressing the language takes priority over progressing the standard library, and this change is on the path towards locking in the final language design with respect to async I/O.

Performance Data Point

0.14.1 untar.zig

conststd=@import("std");
pubfnmain() !void {
constinput_file=trystd.fs.openFileAbsolute("/home/andy/tmp/zig.tar.gz", .{});
varbr=std.io.bufferedReader(input_file.reader());
vardecompress=std.compress.gzip.decompressor(br.reader());
varout_dir=trystd.fs.cwd().makeOpenPath("out", .{});
deferout_dir.close();
deferstd.fs.cwd().deleteTree("out") catch@panic("delete fail");
trystd.tar.pipeToFileSystem(out_dir, decompress.reader(), .{
.strip_components=0,
.mode_mode=.ignore,
.exclude_empty_directories=true,
});
}

this branch untar.zig

conststd=@import("std");
pubfnmain() !void {
constinput_file=trystd.fs.openFileAbsolute("/home/andy/tmp/zig.tar.gz", .{});
varinput_file_buffer: [4096]u8=undefined;
varinput_file_reader=input_file.reader(&input_file_buffer);
varout_dir=trystd.fs.cwd().makeOpenPath("out", .{});
deferout_dir.close();
deferstd.fs.cwd().deleteTree("out") catch@panic("delete fail");
varflate_buffer: [std.compress.flate.max_window_len]u8=undefined;
vardecompress: std.compress.flate.Decompress= .init(&input_file_reader.interface, .gzip, &flate_buffer);
trystd.tar.pipeToFileSystem(out_dir, &decompress.reader, .{
.strip_components=0,
.mode_mode=.ignore,
.exclude_empty_directories=true,
});
}

Benchmark 1 (10 runs): 0.14.1 untar
measurement mean ± σ min … max outliers delta
wall_time 1.62s ± 18.7ms 1.60s … 1.67s 1 (10%) 0%
peak_rss 668KB ± 0 668KB … 668KB 0 ( 0%) 0%
cpu_cycles 5.94G ± 6.25M 5.93G … 5.95G 0 ( 0%) 0%
instructions 8.18G ± 11.9K 8.18G … 8.18G 0 ( 0%) 0%
cache_references 47.6M ± 464K 47.0M … 48.3M 0 ( 0%) 0%
cache_misses 240K ± 45.4K 198K … 330K 0 ( 0%) 0%
branch_misses 43.7M ± 40.7K 43.7M … 43.8M 0 ( 0%) 0%
Benchmark 2 (11 runs): this branch untar
measurement mean ± σ min … max outliers delta
wall_time 1.43s ± 5.23ms 1.42s … 1.44s 0 ( 0%) ⚡- 11.9% ± 0.8%
peak_rss 688KB ± 0 688KB … 688KB 0 ( 0%) 💩+ 3.1% ± 0.0%
cpu_cycles 5.14G ± 13.0M 5.12G … 5.17G 0 ( 0%) ⚡- 13.5% ± 0.2%
instructions 11.0G ± 9.49K 11.0G … 11.0G 0 ( 0%) 💩+ 34.0% ± 0.0%
cache_references 50.2M ± 823K 48.6M … 51.3M 0 ( 0%) 💩+ 5.4% ± 1.3%
cache_misses 305K ± 50.9K 241K … 405K 0 ( 0%) 💩+ 26.9% ± 18.4%
branch_misses 72.4M ± 81.8K 72.3M … 72.6M 0 ( 0%) 💩+ 65.6% ± 0.1%

Some notable factors:

• new code is not calculating a checksum since it can be done out-of-band
• new code has the fancy match logic replaced with a naive for loop. in the future it would be nice to add a memory copying utility for this that zstd would also use.

Compiler Size Data Point (ReleaseSmall)

master branch vs this branch: 13.8 -> 13.5 MiB (-2%)

[dotcarmen]

• delete std.Io.SeekableStream
  • Instead, use *std.fs.File.Reader, *std.fs.File.Writer, or std.ArrayListUnmanaged concrete types, because the implementations will be fundamentally different based on whether you are operating on files or memory.

am I to understand correctly that this means there will be no generic seeking implementation in the new io api? want to confirm before i consider opening a PR

[andrewrk]

am I to understand correctly that this means there will be no generic seeking implementation in the new io api? want to confirm before i consider opening a PR

That's correct. If you need a seekable reader or writer, I recommend to have two API endpoints: one that takes a slice of memory, and one that takes a *File.Reader or *File.Writer. These are fundamentally different use cases that deserve different implementations. However, they can still share code; once the memory slice version calculates the memory offset, it can create a fixed Io.Reader or Io.Writer, and once the seek position has been set in File.Reader/File.Writer, the interface field can be referenced. Then you can have common code that accepts the Io.Reader/Io.Writer interface.