⚡ bufzilla

buffer • zilla

A compact and fast binary encoding format in pure Zig. Originally based on rxi's article - "A Simple Serialization System".

bufzilla is ideal for serializing JSON-like objects and arrays, and has the following qualities:

• Portable across endianness and architectures.
• Schemaless, fully self-describing format; no "pre-compilation" step is necessary.
• Zero-copy reads directly from the encoded bytes.
• Zero internal allocations while encoding/decoding.
• Variable length integer encoding enabled by default, no wasted bytes.
• Safety against untrusted inputs with configurable, zero-overhead parsing limits.
• Data can be read linearly without any intermediate representation (eg. trees).
• Format encoded objects as JSON via Inspect API.
• Serialize Zig structs and data types recursively.

Installation

• Zig version: 0.15.2

zig fetch https://github.com/theseyan/bufzilla/archive/refs/tags/{VERSION}.tar.gz

Copy the hash generated and add bufzilla to your build.zig.zon:

.{
.dependencies= .{
.bufzilla= .{
.url="https://github.com/theseyan/bufzilla/archive/refs/tags/{VERSION}.tar.gz",
.hash="{HASH}",
},
},
}

Usage

bufzilla simply takes a std.Io.Writer interface, and writes encoded data to it. Such a writer can be backed by a growing buffer, a fixed array, a file, or a network socket, etc.

Writing to a dynamic buffer

Use std.Io.Writer.Allocating when you need a dynamically growing buffer:

conststd=@import("std");
constIo=std.Io;
constWriter=@import("bufzilla").Writer;
// Create an allocating writervaraw=Io.Writer.Allocating.init(allocator);
deferaw.deinit();
// Initialize bufzilla writervarwriter=Writer.init(&aw.writer);
constDataType=struct {
a: i64,
b: struct {
c: bool,
},
d: []constunion(enum) {
null: ?void,
f64: f64,
string: []constu8,
},
};
constdata=DataType{
.a=123,
.b= .{ .c=true },
.d= &.{ .{ .f64=123.123 }, .{ .null=null }, .{ .string="value" } },
};
trywriter.writeAny(data);
// Get the encoded bytesconstencoded=aw.written();
std.debug.print("Encoded {d} bytes\n", .{encoded.len});

Writing to a fixed buffer

Use std.Io.Writer.fixed to prevent dynamic allocations when you know the maximum size upfront:

varbuffer: [1024]u8=undefined;
varfixed=Io.Writer.fixed(&buffer);
varwriter=Writer.init(&fixed);
trywriter.writeAny("hello");
trywriter.writeAny(@as(i64, 42));
constencoded=fixed.buffered();

Incremental writing

You can also build messages incrementally:

varwriter=Writer.init(&aw.writer);
trywriter.startObject();
trywriter.writeAny("name");
trywriter.writeAny("Alice");
trywriter.writeAny("scores");
trywriter.startArray();
trywriter.writeAny(@as(i64, 100));
trywriter.writeAny(@as(i64, 95));
trywriter.endContainer(); // end arraytrywriter.endContainer(); // end object

Typed arrays

When all elements of an array have the same numeric type, and the length is known upfront, you can use a packed typed array which is much more space-efficient:

constvec= [_]f32{1.0} **768;
trywriter.writeTypedArray(&vec);

Streaming updates

If you already have an encoded object buffer and want to update a few fields, Writer.applyUpdates can stream a new encoding efficiently in a single pass. Unchanged parts of the buffer are copied directly while only the changed fields are modified.

constbufzilla=@import("bufzilla");
constWriter=bufzilla.Writer;
// `encoded_bytes` is an existing encoded objectvarnew_age: i64=31;
varnew_city: []constu8="Berlin";
varupdates= [_]Writer.Update{
Writer.Update.init("age", &new_age),
Writer.Update.init("profile.address.city", &new_city),
Writer.Update.init("new_key", &true), // upsert missing keysWriter.Update.init("scores[5]", &@as(i64, 100)), // extend arrays, fills gaps with null
};
varout_writer=Writer.init(&aw.writer);
tryout_writer.applyUpdates(encoded_bytes, updates[0..]);

Note:

• Paths follow the same syntax as Reader.readPath.
• Update.init takes a pointer; the pointed-to value must stay alive for the duration of applyUpdates.
• Missing keys/indices are inserted in the output encoding.
• Errors are returned as Writer.Error || ReadError || ApplyUpdatesError.

Inspecting encoded data as JSON

The Inspect API renders encoded bufzilla data as pretty-printed JSON:

constInspect=@import("bufzilla").Inspect;
// Output to an allocating writervaraw=Io.Writer.Allocating.init(allocator);
deferaw.deinit();
varinspector=Inspect(.{}).init(encoded_bytes, &aw.writer, .{});
tryinspector.inspect();
std.debug.print("{s}\n", .{aw.written()});

Or output directly to a fixed buffer:

varbuffer: [4096]u8=undefined;
varfixed=Io.Writer.fixed(&buffer);
varinspector=Inspect(.{}).init(encoded_bytes, &fixed, .{});
tryinspector.inspect();
std.debug.print("{s}\n", .{fixed.buffered()});

Output:

{
"a": 123,
"b": {
"c": true
},
"d": [
123.12300000000000,
null,
"value"
]
}

Reading encoded data

The Reader provides zero-copy access to encoded data:

constReader=@import("bufzilla").Reader;
varreader=Reader(.{}).init(encoded_bytes);
// Read values sequentiallyconstval=tryreader.read();
switch (val) {
.object=> { /*iterateobject*/ },
.array=> { /*iteratearray*/ },
.i64=>|n|std.debug.print("int: {d}\n", .{n}),
.bytes=>|s|std.debug.print("string: {s}\n", .{s}),
// ... other types
}
// Or iterate containerswhile (tryreader.iterateObject(obj)) |kv| {
// kv.key and kv.value
}

Path-based queries

The readPath method provides fast access to nested values using Javascript-style property paths:

constReader=@import("bufzilla").Reader;
varreader=Reader(.{}).init(encoded_bytes);
// Simple property accessconstname=tryreader.readPath("name");
// Nested propertiesconstcity=tryreader.readPath("address.city");
// Array indexingconstfirst=tryreader.readPath("items[0]");
// Mixed pathsconstscore=tryreader.readPath("users[5].scores[0]");
// Quoted keysconstvalue=tryreader.readPath("data['key.with.dots']");
// Returns null if path doesn't existconstmissing=tryreader.readPath("nonexistent.path"); // null

For reading many paths efficiently, use readPaths to scan the buffer once:

varqueries= [_]bufzilla.PathQuery{
.{ .path="name" },
.{ .path="address.city" },
.{ .path="items[0]" },
};
tryreader.readPaths(queries[0..]);
constname=queries[0].value;
constcity=queries[1].value;
constfirst_item=queries[2].value;

You can find more examples in the unit tests.

Safety against untrusted inputs

When reading untrusted data, bufzilla provides configurable limits at compile time to prevent infinite recursion/stack overflow errors, with negligible performance loss.

constReader=@import("bufzilla").Reader;
// Default limitsvarreader=Reader(.{}).init(data);
// Custom limitsvarreader=Reader(.{
.max_depth=50, // Max nesting depth
.max_bytes_length=1024*1024, // Max string/binary blob size
.max_array_length=10_000, // Max array elements
.max_object_size=10_000, // Max object key-value pairs
}).init(data);
// Unlimited depthvarreader=Reader(.{ .max_depth=null }).init(data);

Limit	Default	Error
max_depth	2048	MaxDepthExceeded
max_bytes_length	unlimited	BytesTooLong
max_array_length	unlimited	ArrayTooLarge
max_object_size	unlimited	ObjectTooLarge

Notes:

• max_array_length and max_object_size require max_depth to be set. Setting them with max_depth = null is a compile error.
• Reader internally allocates a stack buffer of size max_depth for iteration counters when array/object limits are enabled. Keep max_depth reasonable (default 2048 uses ~16KB).

The Inspect API also accepts limits as a parameter:

varinspector=Inspect(.{ .max_depth=100 }).init(data, &writer, .{});

Gotchas

• As a self-describing format, field names (keys) are present in the encoded result which can inflate the encoded size compared to other schemaful encoding formats.

Testing

Comprehensive unit tests are present in the test/ directory.

zig build test

Benchmarks

Run the benchmark suite with zig build bench -Doptimize=ReleaseFast.

bufzilla is competitive in performance with similar self-describing binary formats, and in most cases, much faster.Avg. throughput (ops/sec): bufzilla vs zbor vs zig-msgpack

Results on x86_64 Linux, Ryzen 7 9700X CPU:

Basic Types:
--------------------------------------------------------------------------------
Null Write | 1000000 iterations | 0 ns/op | 0 ops/sec
Null Read | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Bool Write | 1000000 iterations | 0 ns/op | 0 ops/sec
Bool Read | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Small Int Write | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Small Int Read | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Large Int Write | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Large Int Read | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Float Write | 1000000 iterations | 2 ns/op | 500000000 ops/sec
Float Read | 1000000 iterations | 1 ns/op | 1000000000 ops/sec
Strings:
--------------------------------------------------------------------------------
Short String Write (5 bytes) | 500000 iterations | 4 ns/op | 250000000 ops/sec
Short String Read (5 bytes) | 500000 iterations | 2 ns/op | 500000000 ops/sec
Medium String Write (~300 bytes) | 100000 iterations | 4 ns/op | 250000000 ops/sec
Medium String Read (~300 bytes) | 100000 iterations | 2 ns/op | 500000000 ops/sec
Binary Data:
--------------------------------------------------------------------------------
Small Binary Write (32 bytes) | 500000 iterations | 6 ns/op | 166666666 ops/sec
Small Binary Read (32 bytes) | 500000 iterations | 2 ns/op | 500000000 ops/sec
Large Binary Write (1KB) | 100000 iterations | 9 ns/op | 111111111 ops/sec
Large Binary Read (1KB) | 100000 iterations | 2 ns/op | 500000000 ops/sec
Arrays:
--------------------------------------------------------------------------------
Small Array Write (10 elements) | 100000 iterations | 25 ns/op | 40000000 ops/sec
Small Array Read (10 elements) | 100000 iterations | 27 ns/op | 37037037 ops/sec
Medium Array Write (100 elements) | 50000 iterations | 260 ns/op | 3846153 ops/sec
Medium Array Read (100 elements) | 50000 iterations | 243 ns/op | 4115226 ops/sec
Objects (Maps):
--------------------------------------------------------------------------------
Small Object Write (10 entries) | 100000 iterations | 143 ns/op | 6993006 ops/sec
Small Object Read (10 entries) | 100000 iterations | 116 ns/op | 8620689 ops/sec
Medium Object Write (50 entries) | 50000 iterations | 712 ns/op | 1404494 ops/sec
Medium Object Read (50 entries) | 50000 iterations | 629 ns/op | 1589825 ops/sec
Complex Structures:
--------------------------------------------------------------------------------
Nested Structure Write | 50000 iterations | 36 ns/op | 27777777 ops/sec
Nested Structure Read | 50000 iterations | 71 ns/op | 14084507 ops/sec
Mixed Types Write | 50000 iterations | 31 ns/op | 32258064 ops/sec
Mixed Types Read | 50000 iterations | 57 ns/op | 17543859 ops/sec
Struct Serialization:
--------------------------------------------------------------------------------
Simple Struct Write | 100000 iterations | 29 ns/op | 34482758 ops/sec
Simple Struct Read | 100000 iterations | 46 ns/op | 21739130 ops/sec
Complex Struct Write | 50000 iterations | 136 ns/op | 7352941 ops/sec
Complex Struct Read | 50000 iterations | 272 ns/op | 3676470 ops/sec