This package contains a high-performance binary serializer for Go that encodes and decodes arbitrary data structures of variable size. It is designed as a simple json.Marshal / json.Unmarshal drop-in for cases where you control both ends and want compact, fast payloads — for example inter-broker message encoding in emitter.
- Simple API that mirrors
encoding/jsonwithMarshal,Unmarshal, andMarshalTo. - Compact payloads using varint encoding for integers and size-prefixed variable-length values.
- Zero-allocation encoding path via
MarshalTowriting directly to anio.Writer. - Reflect-based support for structs, maps, slices, arrays, pointers, and nested types.
- Fast paths for
[]byteand other common slice types. - Custom serialization via
encoding.BinaryMarshaler/BinaryUnmarshaleror a fullCodecthroughGetBinaryCodec. - Field skipping with the
binary:"-"struct tag. - Tagged unions (oneof / versioning) via
binary:"N,union"tags on pointer fields. - Optional subpackages for sorted, unsafe, and nocopy typed slices when you need smaller payloads or lower decode cost.
Variable-sized values are prefixed with a varint-encoded size and encoded recursively. The default sequential format is not self-describing or automatically versioned; tagged unions provide explicit, user-managed versioning. The format is Go-specific and intended for systems you control.
- Quick Start
- Streaming Encode and Decode
- Skipping Fields
- Tagged Unions
- Custom Serialization
- Typed Slice Subpackages
- Benchmarks
- Disclaimer
- Contributing
- License
Define a message and marshal it the same way you would with JSON:
typemessagestruct {
NamestringTimestampint64Payload []byteSsid []uint32
}
v:=&message{
Name: "Roman",
Timestamp: 1242345235,
Payload: []byte("hi"),
Ssid: []uint32{1, 2, 3},
}
encoded, err:=binary.Marshal(v)
iferr!=nil {
panic(err)
}
varoutmessageerr=binary.Unmarshal(encoded, &out)For hot paths, write into a reused buffer with MarshalTo, or use Encoder / Decoder directly against an io.Writer / io.Reader:
varbuf bytes.Bufferiferr:=binary.MarshalTo(v, &buf); err!=nil {
panic(err)
}
dec:=binary.NewDecoder(&buf)
varoutmessageiferr:=dec.Decode(&out); err!=nil {
panic(err)
}Fields tagged with binary:"-" are ignored during encode and decode. Useful for locks, caches, or derived state:
typeCachestruct {
mu sync.Mutex`binary:"-"`KeystringValue []byte
}A struct whose included fields all use binary:"N,union" tags (N in 1..255) is encoded as a tagged union (oneof / versioning):
uvarint(tag) + uvarint(len) + body
Arms must be pointers and tags must be unique. Exactly one may be non-nil on encode; all nil writes tag 0 with an empty body. Unknown tags are skipped on decode (forward-compatible versioning).
typePayloadstruct {
Text*TextPayload`binary:"1,union"`Image*ImagePayload`binary:"2,union"`
}
encoded, err:=binary.Marshal(Payload{Text: &TextPayload{Msg: "hi"}})
varoutPayloaderr=binary.Unmarshal(encoded, &out)
// out.Text != nilUse one arm per schema version. Older readers ignore newer tags; newer readers still decode older payloads:
typeDocV1struct {
Titlestring
}
typeDocV2struct {
TitlestringBodystring
}
typeDocstruct {
V1*DocV1`binary:"1,union"`V2*DocV2`binary:"2,union"`
}
varErrUnknownVersion=errors.New("unknown document version")
// Current returns the latest representation, migrating older versions explicitly.func (dDoc) Current() (DocV2, error) {
switch {
cased.V2!=nil:
return*d.V2, nilcased.V1!=nil:
returnDocV2{Title: d.V1.Title}, nildefault:
returnDocV2{}, ErrUnknownVersion
}
}
// Writer on the new versionencoded, err:=binary.Marshal(Doc{V2: &DocV2{Title: "hi", Body: "world"}})
// Reader that only knows V1 still succeeds: unknown tag 2 is skippedvarlegacystruct {
V1*DocV1`binary:"1,union"`
}
err=binary.Unmarshal(encoded, &legacy)
// legacy.V1 == nil — body was skipped, no error// Reader that knows both picks the matching armvarcurrentDocerr=binary.Unmarshal(encoded, ¤t)
latest, err:=current.Current()Nest a union inside a normal sequential struct as a single field. For hand-rolled codecs, use Encoder.WriteTagged / Decoder.ReadTagged with the same framing.
By default, values are encoded through reflection. You can override that for a type in two ways, checked in this order:
GetBinaryCodec()— return abinary.Codecfor full control over the wire format (no extra length prefix).MarshalBinary/UnmarshalBinary— the standardencoding.BinaryMarshaler/BinaryUnmarshalerpair; the package length-prefixes the returned bytes for you.
Use MarshalBinary when you already have a []byte representation. Use GetBinaryCodec when you want to stream fields through the encoder without an intermediate buffer.
This matches encoding.BinaryMarshaler and encoding.BinaryUnmarshaler. On encode, the returned slice is written as uvarint(length) + bytes. On decode, that framed blob is passed to UnmarshalBinary:
typeCompactHeaderstruct {
Codeuint8
}
func (hCompactHeader) MarshalBinary() ([]byte, error) {
return []byte{h.Code}, nil
}
func (h*CompactHeader) UnmarshalBinary(data []byte) error {
iflen(data) !=1 {
returnfmt.Errorf("CompactHeader: want 1 byte, got %d", len(data))
}
h.Code=data[0]
returnnil
}
encoded, err:=binary.Marshal(CompactHeader{Code: 0x13})
// encoded == []byte{0x01, 0x13} // length + payloadvaroutCompactHeadererr=binary.Unmarshal(encoded, &out)This is enough for most custom types, including anything that already implements the standard library interfaces (for example time.Time).
For tighter packing or to avoid the intermediate []byte, implement GetBinaryCodec on a pointer receiver. It must return a binary.Codec:
typeCodecinterface {
EncodeTo(*Encoder, reflect.Value) errorDecodeTo(*Decoder, reflect.Value) error
}Example: encode a 2D point as two raw float64 values with no struct field metadata:
typePointstruct {
Xfloat64Yfloat64
}
func (p*Point) GetBinaryCodec() binary.Codec {
returnpointCodec{}
}
typepointCodecstruct{}
func (pointCodec) EncodeTo(e*binary.Encoder, rv reflect.Value) error {
p:=rv.Interface().(Point)
e.WriteFloat64(p.X)
e.WriteFloat64(p.Y)
returnnil
}
func (pointCodec) DecodeTo(d*binary.Decoder, rv reflect.Value) error {
x, err:=d.ReadFloat64()
iferr!=nil {
returnerr
}
y, err:=d.ReadFloat64()
iferr!=nil {
returnerr
}
rv.Set(reflect.ValueOf(Point{X: x, Y: y}))
returnnil
}Because GetBinaryCodec takes priority, a type should not also rely on MarshalBinary for the same purpose — pick one approach per type.
Nested use works automatically: if a struct field's type implements either customization, that field uses the custom path while the rest of the struct uses the default reflect codecs.
typePacketstruct {
IDuint64At time.Time// uses time.Time's BinaryMarshalerPosPoint// uses GetBinaryCodec aboveRaw []byte
}Optional helpers live in subpackages when the default reflect path is not enough:
| Package | Purpose |
|---|---|
sorted | Delta-encoded sorted integer / timestamp slices for smaller wire size |
unsafe | Memory-cast numeric slices (faster, not portable across endianness) |
nocopy | Like unsafe, but decode reuses the input buffer (zero-copy; lifetime tied to the buffer) |
import (
"github.com/kelindar/binary""github.com/kelindar/binary/sorted"
)
v:= sorted.Int32s{4, 5, 6, 1, 2, 3}
encoded, err:=binary.Marshal(&v)
varout sorted.Int32serr=binary.Unmarshal(encoded, &out)See each subpackage README for trade-offs and warnings.
Numbers from bench/. Run locally with:
cd bench && go run .name time/op ops/s allocs/op
-------------------- ------------ ------------ ------------
binary/enc 122.6 ns 8.2M 2
binary/enc-to 89.0 ns 11.2M 0
binary/dec 90.1 ns 11.1M 1
binary/map-enc 8.5 µs 117.4K 209
binary/map-dec 12.8 µs 78.1K 500
binary/slice-enc 8.0 µs 124.3K 9
binary/slice-dec 6.6 µs 151.4K 100
binary/nest-enc 4.2 µs 236.1K 13
binary/nest-dec 3.7 µs 270.0K 63
binary/bytes-enc 812.2 ns 1.2M 3
binary/bytes-dec 172.5 ns 5.8M 0
binary/u64-enc 42.4 µs 23.6K 3
binary/u64-dec 57.6 µs 17.4K 0
binary/reuse-enc 95.7 ns 10.5M 0
binary/stream-dec 147.4 ns 6.8M 2
binary/trace-enc 12.6 µs 79.2K 9
binary/trace-dec 13.0 µs 76.7K 640
union/enc 196.6 ns 5.1M 3
union/dec 100.7 ns 9.9M 0
union/nest-enc 213.3 ns 4.7M 3
union/nest-dec 110.4 ns 9.1M 0
union/reuse-enc 98.3 ns 10.2M 0
union/stream-dec 132.9 ns 7.5M 1
nocopy/str-enc 107.6 ns 9.3M 3
nocopy/str-dec 34.8 ns 28.8M 0
nocopy/dict-enc 183.7 ns 5.4M 2
nocopy/dict-dec 155.5 ns 6.4M 2
nocopy/bmap-enc 322.4 ns 3.1M 5
nocopy/bmap-dec 161.9 ns 6.2M 2
nocopy/hmap-enc 320.1 ns 3.1M 5
nocopy/hmap-dec 143.9 ns 7.0M 2
nocopy/bytes-enc 841.1 ns 1.2M 3
nocopy/bytes-dec 35.8 ns 28.0M 0
nocopy/u64-enc 5.4 µs 186.9K 3
nocopy/u64-dec 32.0 ns 31.2M 0
nocopy/col-enc 504.2 ns 2.0M 8
nocopy/col-dec 359.0 ns 2.8M 6
nocopy/struct-enc 161.0 ns 6.2M 3
nocopy/struct-dec 72.4 ns 13.8M 0
sorted/i32-enc 81.7 µs 12.2K 5
sorted/i32-dec 57.6 µs 17.4K 0
sorted/u32-enc 78.8 µs 12.7K 5
sorted/u32-dec 51.3 µs 19.5K 0
sorted/ts-enc 51.4 µs 19.4K 6
sorted/ts-dec 22.8 µs 43.9K 2
sorted/tsz-enc 129.7 µs 7.7K 7
sorted/tsz-dec 132.0 µs 7.6K 3
sorted/tcz-enc 86.9 µs 11.5K 6
sorted/tcz-dec 79.2 µs 12.6K 3
unsafe/u64-enc 483.4 ns 2.1M 3
unsafe/u64-dec 463.0 ns 2.2M 2
This is not a replacement for JSON, protobuf, or other versioned interchange formats. The codec does not maintain schema evolution or cross-language compatibility. Use it to exchange binary data of a known format between Go services where you control both ends.
Contributions are welcome — open a pull request and we will review it as quickly as we can. This library is maintained by Roman Atachiants.
Binary is licensed under the MIT License.