ProtoCache

ProtoCache is an alternative flat binary format for Protobuf schemas. Like FlatBuffers, it supports direct reads without first materializing an object graph, while usually producing smaller data and supporting maps. The benchmark compares data size, traversal, reflection, and compression costs. ProtoCache is designed for workloads that need a balance between compact cached data and fast reads.

	Protobuf	ProtoCache	FlatBuffers	Cap'n Proto	Fory
Data Size	574B	780B	1296B	1288B	615B
Decode + Traverse + Dealloc	2620ns	132ns	89ns	610ns	1748ns
Decode + Traverse(reflection) + Dealloc	8189ns	270ns	484ns	8748ns	-
Compressed/Packed Size	566B	571B	856B	626B	611B
Compress	258ns	427ns	775ns	-	295ns
Decompress/Unpack	114ns	229ns	532ns	653ns	140ns

A naive compress algorithm is introduced to reduce continuous 0x00 or 0xff bytes, which makes the final output size of ProtoCache close to Protobuf. Because Cap'n Proto has a builtin pack algorithm, which shows better compress ratio than our naive compress algorithm, without explicit compress/decompress API, we take the time gap between access in plain and packed mode as decompress time.

Difference to Protobuf

ProtoCache reserves a single repeated field named _ with field number 1 as an alias to an array or map rather than a normal one-field message. This makes multidimensional containers possible without an extra message layer:

messageVec2D {
messageVec1D {
repeatedfloat_=1;
}
repeatedVec1D_=1;
}

For all other schema and object-model differences, including field numbering, presence and defaults, oneof, maps, deprecated declarations, unknown fields, and evolution rules, see schema.md.

Build and Install

ProtoCache requires a C++17 compiler and Protobuf. Tests additionally require GoogleTest, while command-line tools require gflags. CMake 3.13 or newer is required.

cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DWITH_TEST=ON \
-DWITH_TOOLS=ON
cmake --build build
ctest --test-dir build --output-on-failure
cmake --install build --prefix /path/to/prefix

WITH_BENCHMARK and PROTOCACHE_ENABLE_NATIVE_OPT are disabled by default so normal builds remain portable. Benchmark builds enable native CPU optimization and require the additional benchmark libraries used by this repository. PROTOCACHE_BUILD_SHARED is enabled by default on Linux and macOS and disabled on Windows. Windows currently supports the static libraries, tools, and Python extension; building the shared library there is not supported yet.

Installed CMake packages can be consumed directly:

find_package(ProtoCache1.2CONFIGREQUIRED)
target_link_libraries(my_targetPRIVATEProtoCache::protocache)

Available library targets are ProtoCache::protocache (full static library) and ProtoCache::protocache-lite (static core library without the reflection extension). When PROTOCACHE_BUILD_SHARED=ON, the install also provides ProtoCache::protocache-shared on Linux and macOS. When tools are enabled, the install contains the JSON converters and all protoc plugins.

Format compatibility is governed by data-format.md.

Code Gen

protoc --pccx_out=. [--pccx_opt=extra] test.proto

A protobuf compiler plugin called protoc-gen-pccx is available to generate header-only C++ file. If option extra is set, it will generate another file for extra APIs.

APIs

protocache::Buffer buf1;
ASSERT_TRUE(protocache::Serialize(pb_message, &buf1));
// =========basic api=========auto& root = protocache::Message(buf1.View()).Cast<test::Main>();
ASSERT_FALSE(!root);
// =========extra api=========
::ex::test::Main ex_root(buf1.View());
protocache::Buffer buf2;
ASSERT_TRUE(ex_root.Serialize(&buf2));
ASSERT_EQ(buf1.Size(), buf2.Size());
// deserialize to pbprotocache::Deserialize(data, &pb_mirror);

You can create protocache binary by serializing a protobuf message with protocache::Serialize. The Basic API offers fast read-only access with zero-copy technique. Extra APIs provide a mutable object and another serialization method, which only reserialize accessed parts.

	Protobuf	ProtoCacheEX	ProtoCache
Serialize	557ns	308 ~ 1879ns	6493ns
Decode + Traverse + Dealloc	2620ns	1227ns	132ns
Serialize (twitter.proto)	215us	101us	412us

Test full serialization with a complicated twitter.proto. Since serializing big object is a memory-bound task, ProtoCache may show it's advantage.

Reflection

std::string err;
google::protobuf::FileDescriptorProto file;
ASSERT_TRUE(protocache::ParseProtoFile("test.proto", &file, &err));
protocache::reflection::DescriptorPool pool;
ASSERT_TRUE(pool.Register(file));
auto descriptor = pool.Find("test.Main");
ASSERT_NE(descriptor, nullptr);

The reflection apis are simliar to protobuf's. An example can be found in the test. If you don't need reflection including the basic serialize API, linking protocache-lite instead of protocache library to avoid dependency on protobuf may be a good idea.

Other Implements

Language	Source
Python	python
TypeScript	typescript
Go	https://github.com/peterrk/protocache-go
Java	https://github.com/peterrk/protocache-java
C#	https://github.com/peterrk/protocache.net
Rust	https://github.com/peterrk/protocache-rust
Swift	https://github.com/peterrk/protocache-swift

Data Size Evaluation

Following work in paper, we can find that protocache has smaller data size than FlatBuffers and Cap'n Proto, in most cases.

	Protobuf	ProtoCache	FlatBuffers	Cap'n Proto	ProtoCache (Packed)	Cap'n Proto (Packed)
CircleCI Definition (Blank)	5	8	20	24	6	6
CircleCI Matrix Definition	26	88	104	96	50	36
Entry Point Regulation Manifest	247	352	504	536	303	318
ESLint Configuration Document	161	276	320	216	175	131
ECMAScript Module Loader Definition	23	44	80	80	33	35
GeoJSON Example Document	325	432	680	448	250	228
GitHub FUNDING Sponsorship Definition (Empty)	17	24	68	40	23	25
GitHub Workflow Definition	189	288	440	464	237	242
Grunt.js Clean Task Definition	20	48	116	96	28	39
ImageOptimizer Azure Webjob Configuration	23	60	100	96	40	44
JSON-e Templating Engine Reverse Sort Example	21	68	136	240	38	43
JSON-e Templating Engine Sort Example	10	36	44	48	21	18
JSON Feed Example Document	413	484	584	568	474	470
JSON Resume Example	2225	2608	3116	3152	2537	2549
.NET Core Project	284	328	636	608	303	376
OpenWeatherMap API Example Document	188	244	384	320	199	206
OpenWeather Road Risk API Example	173	240	328	296	205	204
NPM Package.json Example Manifest	1581	1736	2268	2216	1726	1755
TravisCI Notifications Configuration	521	600	668	640	601	566
TSLint Linter Definition (Basic)	8	24	60	48	14	12
TSLint Linter Definition (Extends Only)	47	68	88	88	62	62
TSLint Linter Definition (Multi-rule)	14	32	84	80	20	23