Marshal and Unmarshal Binary Formats in Go

The Go standard library features packages for converting between structs and various message and file formats. Packages encoding/json and encoding/xml are well known for providing convenient functions Marshal() and Unmarshal() that leverage the declarative nature of struct tags. These functions are however missing from encoding/binary, leaving developers without an accessible and intuitive way to work with binary formats.

This module supplies the ubiquitous Marshal() and Unmarshal() function signatures so that developers can define custom binary formats using only struct tags and avoid custom bit manipulation.

User Story

As a Go developer implementing a binary message or file format,
I want a pair of functions "Marshal/Unmarshal" like those in "encoding/json"
that convert a struct into a series of bits in a byte slice and vice versa,
so that I can avoid the complexities of custom bit manipulation.

See the rest of the behaviour-driven specifications of this module below.

Binary Formats

Message and file formats specify how bits are arranged to encode information. Control over individual bits or groups smaller than a byte is often required to put together and take apart these binary structures.

Message Format Examples

Describing the anatomy of TCP/IP headers at the beginning of every internet datagram ("packet") are

• Section 3.1 of RFC 791 Internet Protocol, and
• Section 3.1 of RFC 793 Transmission Control Protocol.

File Format Examples

Binary file formats are not significantly different from message formats from an application developer's perspective. RFC 1952 describes the GZIP File Format Specification.

Working with Binary Formats in Go

The smallest data structures Go provides are the basic type byte (alias of uint8, an unsigned 8-bit integer), and bool, both eight bits long. To manipulate data at a scale smaller than eight bits would require the use of bitwise logical and shift operators such as AND (&), OR (|), left shift (<<), and right shift (>>).

Relevant Questions Posted on StackOverflow

Suggestions on StackOverflow are limited to the use of bitwise operators.

• Golang: Parse bit values from a byte
• Creating 8 bit binary data from 4,3, and 1 bit data in Golang
• How to pack the C bit field struct via encoding package in GO?

Behaviour-Driven Specifications

Feature: Marshal and Unmarshal
As a Go developer implementing a binary message or file format,
I want a pair of functions "Marshal/Unmarshal" like those in "encoding/json"
that convert a struct into a series of bits in a byte slice and vice versa,
so that I can avoid the complexities of custom bit manipulation.
Background:
 # Ubiquitous languageGiven a message or file "format"""" A format specifies how bits are arranged to encode information. """And the format is a series of "bit fields"""" A bit field is one or more adjacent bits representing a value, and should not be confused with struct fields. """And adjacent bit fields are grouped into "words"""" A word is a series of bits that can be simultaneously processed by a given computer architecture and programming language. """ # Define format-structsAnd a format is represented by a type definition of a "format-struct"And the format-struct nests one or more exported "word-structs"And the words are tagged to indicate their lengths in number of bits

typeRFC791InternetHeaderFormatWithoutOptionsstruct {
RFC791InternetHeaderFormatWord0`word:"32"`RFC791InternetHeaderFormatWord1`word:"32"`RFC791InternetHeaderFormatWord2`word:"32"`RFC791InternetHeaderFormatWord3`word:"32"`RFC791InternetHeaderFormatWord4`word:"32"`
}

And the length of each word is a multiple of eight in the range [8, 64]
 # Define word-structsAnd each word-struct has exported field(s) corresponding to bit field(s)
And the fields are of unsigned integer or boolean types
And the fields are tagged to indicate the lengths of those bit fields

typeRFC791InternetHeaderFormatWord0struct {
Versionuint8`bitfield:"4"`IHLuint8`bitfield:"4"`Precedenceuint8`bitfield:"3"`Delaybool`bitfield:"1"`Throughputbool`bitfield:"1"`Reliabilitybool`bitfield:"1"`Reserveduint8`bitfield:"2"`TotalLengthuint16`bitfield:"16"`
}

And the length of each bit field does not overflow the type of the field
""" A bit field overflows a type when it is long enough to represent values outside the set of values of the type. """And the sum of lengths of all fields is equal to the length of that word

Marshal

Scenario: Marshal a struct into a byte sliceGiven a format-struct variable representing a binary message or file

internetHeader=RFC791InternetHeaderFormatWithoutOptions{
RFC791InternetHeaderFormatWord0{
Version: 4,
IHL: 5,
// ...
},
// ...
}

And the struct field values do not overflow corresponding bit fields
""" A struct field value overflows its corresponding bit field when it falls outside the range of values that can be represented by that bit field given its length. """When I pass to function Marshal() a pointer to that struct variable

var (
bytes []byteeerror
)
bytes, e=binary.Marshal(&internetHeader)

Then Marshal() should return a slice of bytes and a nil error
And I should see struct field values reflected as bits in those bytes

log.Printf("%08b", bytes)
// [01000101 ...]log.Println(e==nil)
// true

And I should see that the lengths of the slice and the format are equal
""" The length of a format is the sum of lengths of the words in it. The length of a word is the sum of lengths of the bit fields in it. """

Unmarshal

Scenario: Unmarshal a byte slice into a structGiven a format-struct type representing a binary message or file format

varinternetHeaderRFC791InternetHeaderFormatWithoutOptions

And a slice of bytes containing a binary message or file

varbytes []byte// ...log.Printf("%08b", bytes)
// [01000101 ...]

And the lengths of the slice and the format (measured in bits) are equal
When I pass to function Unmarshal() the slice of bytes as an argument
And I pass to the function a pointer to the struct as a second argument

e=binary.Unmarshal(bytes, &internetHeader)

Then Unmarshal() should return a nil error
And I should see struct field values matching the bits in those bytes

log.Println(e==nil)
// truelog.Println(internetHeader.RFC791InternetHeaderFormatWord0.Version)
// 4log.Println(internetHeader.RFC791InternetHeaderFormatWord0.IHL)
// 5

Performance and Optimisation

This module is optimised for performance.

$ go test -cpuprofile cpu.prof -memprofile mem.prof -bench . -benchmem

goos: linux
goarch: arm64
pkg: github.com/encodingx/binary
BenchmarkMarshal-2 3435181 349.9 ns/op 64 B/op 6 allocs/op
BenchmarkUnmarshal-2 3004425 396.4 ns/op 64 B/op 8 allocs/op
PASS
ok github.com/encodingx/binary	3.288s