Protarrow

Protarrow is a python library for converting from Protocol Buffers to Apache Arrow and back.

It is used at Tradewell Technologies, to share data between transactional and analytical applications, with little boilerplate code and zero data loss.

Installation

pip install protarrow

Usage

Taking a simple protobuf:

messageMyProto {
stringname=1;
int32id=2;
repeatedint32values=3;
}

It can be converted to a pyarrow.Table:

importprotarrowmy_protos= [
MyProto(name="foo", id=1, values=[1, 2, 4]),
MyProto(name="bar", id=2, values=[3, 4, 5]),
]
table=protarrow.messages_to_table(my_protos, MyProto)

name	id	values
foo	1	[1 2 4]
bar	2	[3 4 5]

And the table can be converted back to proto:

protos_from_table=protarrow.table_to_messages(table, MyProto)

See the documentation