obj2buf

A type-safe encoder/decoder for structured binary data with snake_case API design.

Transform JavaScript objects into compact binary buffers and back with zero data loss. Suitable for network protocols, file formats, and high-performance data exchange.

Features

• Simple API: Clean snake_case interface with MapType for structured data
• High Performance: Efficient little-endian encoding with optional unsafe mode (4-5x faster)
• Type Safety: Comprehensive type system with strict validation
• Variable Length Support: VarStringType and variable-length arrays with automatic header optimization
• JSON Serialization: Full schema persistence with consistent naming
• Well Documented: Complete JSDoc documentation for IDE support
• Tested: 243+ tests covering real-world scenarios
• Cross-Language: C++ bindings available for native applications

Versioning

• JavaScript Library: Follows semantic versioning (semver)
• C++ Bindings: Independent versioning, compatibility noted in documentation
• Binary Format: Stable across patch versions, changes documented in major/minor releases

Installation

JavaScript/Node.js

npm install obj2buf

C++ Bindings

The package includes C++11 header-only bindings for deserializing data in native applications. See bindings/cpp/README.md for details.

Current version: C++ bindings v1.0.0 (compatible with obj2buf JavaScript v1.0.0+)

Quick Start

const{ Schema, types }=require('obj2buf');// Define structured data using MapTypeconstuser_type=newtypes.MapType([['id',newtypes.UInt32()],['username',newtypes.FixedStringType(20)],['email',newtypes.VarStringType(100)],['age',newtypes.UInt8()],['is_active',newtypes.BooleanType()],['score',newtypes.OptionalType(newtypes.Float32())]]);// Create schema with the typeconstuser_schema=newSchema(user_type);// Your dataconstuser_data={id: 12345,username: 'john_doe',email: 'john@example.com',age: 28,is_active: true,score: 95.5};// Serialize to binaryconstbuffer=user_schema.serialize(user_data);console.log('Encoded size:',buffer.length,'bytes');// Deserialize back to objectconstdecoded=user_schema.deserialize(buffer);console.log('Decoded:',decoded);

Type System

Primitive Types

• UInt8, UInt16, UInt32 - Unsigned integers (1, 2, 4 bytes)
• Int8, Int16, Int32 - Signed integers (1, 2, 4 bytes)
• Float32, Float64 - IEEE 754 floating point (4, 8 bytes)
• BooleanType - True/false values (1 byte)
• Char - Single UTF-8 character (1 byte)

Unified Types (Alternative Syntax)

• UInt(bytes), Int(bytes), Float(bytes) - Generic constructors

String Types

• FixedStringType(length) - Fixed-length strings with null padding
• VarStringType(max_length?) - Variable-length strings with automatic header optimization
  • Uses 1-byte header for max_length < 256
  • Uses 2-byte header for max_length ≥ 256
  • Default max_length: 65535

Container Types

• ArrayType(element_type, length?) - Arrays (fixed or variable length)
• TupleType(...element_types) - Fixed-structure tuples
• MapType(field_pairs) - Structured objects with named fields
• EnumType(options) - String enumerations with automatic sizing
• OptionalType(base_type) - Nullable values with presence flag

Advanced Examples

Complex Nested Structures

const{ Schema, types }=require('obj2buf');// Game state with nested structuresconstgame_state_type=newtypes.MapType([['player_id',newtypes.UInt32()],['position',newtypes.TupleType(newtypes.Float32(),newtypes.Float32())],['health',newtypes.UInt8()],['inventory',newtypes.ArrayType(newtypes.UInt16(),10)],['weapon',newtypes.EnumType(['sword','bow','staff','dagger'])],['magic_points',newtypes.OptionalType(newtypes.UInt16())],['metadata',newtypes.MapType([['version',newtypes.UInt8()],['timestamp',newtypes.UInt32()],['notes',newtypes.VarStringType(500)]])]]);constgame_schema=newSchema(game_state_type);constgame_state={player_id: 1337,position: [123.45,678.90],health: 85,inventory: [1001,1002,1003,0,0,0,0,0,0,0],weapon: 'sword',magic_points: 150,metadata: {version: 2,timestamp: 1632847200,notes: 'Player reached level 45'}};constbuffer=game_schema.serialize(game_state);constdecoded=game_schema.deserialize(buffer);

Variable-Length Data

// Dynamic arrays and stringsconstdynamic_type=newtypes.MapType([['message',newtypes.VarStringType(1000)],['tags',newtypes.ArrayType(newtypes.VarStringType(50))],// Variable-length array['scores',newtypes.ArrayType(newtypes.UInt16(),20)],// Fixed-length array['metadata',newtypes.OptionalType(newtypes.VarStringType(200))]]);constdynamic_schema=newSchema(dynamic_type);constdata={message: 'Hello, world!',tags: ['important','user-generated','reviewed'],// Any length arrayscores: [100,95,87,92,88,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],metadata: 'Created by automated system'};// Calculate exact size neededconstneeded_bytes=dynamic_schema.calculate_byte_length(data);console.log('Needs',needed_bytes,'bytes');constbuffer=dynamic_schema.serialize(data);

Performance Mode

For high-throughput scenarios, skip validation for maximum speed:

// Up to 5x faster encodingconstfast_buffer=schema.serialize(data,0,{unsafe: true});// ⚠️ WARNING: Only use unsafe mode when you guarantee:// - No null/undefined values for required fields// - Correct data types// - Valid enum values// - String lengths within bounds

Advanced Usage

Choosing the Right API

High-Level Methods (Recommended):

• serialize(data) - Creates and returns a buffer automatically
• deserialize(buffer) - Returns the decoded value directly
• Use these for most applications - they're simple and handle memory management

Low-Level Methods (For Performance/Control):

• encode(data, buffer, offset) - Write to an existing buffer, returns bytes written
• decode(buffer, offset) - Returns {value, bytes_read} for precise control
• Use these when you need to manage your own buffers or process streams

Buffer Management Examples

constschema=newSchema({name: 'string',age: 'uint8'});constdata={name: 'Alice',age: 30};// High-level: Simple and cleanconstbuffer=schema.serialize(data);constdecoded=schema.deserialize(buffer);// Low-level: Manual buffer controlconstmy_buffer=Buffer.alloc(100);constbytes_written=schema.encode(data,my_buffer,0);constresult=schema.decode(my_buffer,0);console.log('Value:',result.value,'Read:',result.bytes_read,'bytes');

Streaming Operations

// High-level: Each message gets its own bufferconstmessages=[{type: 'login',user: 'alice'},{type: 'message',text: 'Hello!'},{type: 'logout',user: 'alice'}];constbuffers=messages.map(msg=>message_schema.serialize(msg));constdecoded=buffers.map(buf=>message_schema.deserialize(buf));// Low-level: Pack multiple messages into one bufferconststream_buffer=Buffer.alloc(1024);letoffset=0;for(constmessageofmessages){offset+=message_schema.encode(message,stream_buffer,offset);}// Read them backoffset=0;constdecoded_messages=[];while(offset<stream_buffer.length){constresult=message_schema.decode(stream_buffer,offset);if(result.bytes_read===0)break;// End of datadecoded_messages.push(result.value);offset+=result.bytes_read;}

Schema Properties & Methods

Properties (snake_case API)

// Static analysisschema.byte_length// Total bytes (null if variable-length)schema.is_static_length// true if all fields are fixed-length// Dynamic calculationschema.calculate_byte_length(data)// Exact bytes needed for specific data

Encoding & Decoding

// High-level methods (recommended)constbuffer=schema.serialize(data)// Auto-allocate bufferconstbuffer=schema.serialize(data,offset)// With offset paddingconstvalue=schema.deserialize(buffer)// Direct valueconstvalue=schema.deserialize(buffer,offset)// With offset// Low-level methods (for performance/control)constbytes=schema.encode(data,buffer,offset)// Requires buffer, returns bytes writtenconst{ value, bytes_read }=schema.decode(buffer,offset)// Returns wrapped result

JSON Serialization

// Export schemaconstjson=schema.to_json()// Import schema constnew_schema=Schema.from_json(json)// Type consistency: obj.type matches class names// ArrayType ↔ "ArrayType" (not "Array")// FixedStringType ↔ "FixedStringType" (not "String")// etc.

Type Features

Automatic Header Optimization

// VarStringType automatically chooses header sizenewtypes.VarStringType(100)// 1-byte header (max < 256)newtypes.VarStringType(1000)// 2-byte header (max ≥ 256)

Enum Auto-Sizing

// EnumType automatically chooses storage sizenewtypes.EnumType(['a','b'])// 1 byte (2 options)newtypes.EnumType([...Array(300).keys()])// 2 bytes (300 options)newtypes.EnumType([...Array(70000).keys()])// 4 bytes (70k options)

Empty Type Support

// Edge cases handled gracefullynewtypes.MapType([])// Empty mapnewtypes.TupleType()// Empty tuplenewtypes.ArrayType(type,0)// Zero-length array

Error Handling

All validation errors throw ParserError with descriptive messages:

const{ ParserError }=require('obj2buf');try{schema.encode({username: null});// Required field is null}catch(error){console.log(errorinstanceofParserError);// trueconsole.log(error.message);// "Cannot encode null as FixedStringType"}

Real-World Example: API Message Format

constapi_message_type=newtypes.MapType([['version',newtypes.UInt8()],['message_type',newtypes.EnumType(['request','response','error','notification'])],['correlation_id',newtypes.VarStringType(36)],// UUID length['timestamp',newtypes.UInt32()],['payload_size',newtypes.UInt32()],['payload',newtypes.VarStringType(1048576)],// 1MB max['headers',newtypes.ArrayType(newtypes.TupleType(newtypes.VarStringType(100),// keynewtypes.VarStringType(500)// value))],['signature',newtypes.OptionalType(newtypes.FixedStringType(64))]]);constmessage_schema=newSchema(api_message_type);// Usage in APIfunctionserialize_message(msg){returnmessage_schema.encode(msg);}functiondeserialize_message(buffer){returnmessage_schema.decode(buffer).value;}

Testing

Run the comprehensive test suite:

npm test

Coverage includes:

• 243+ tests with complete coverage
• All primitive and complex types
• Edge cases and error conditions
• Real-world usage scenarios
• JSON serialization round-trips
• Performance benchmarks
• Memory efficiency tests

License

ISC