Native model

Add interoperability on the top of serialization formats like bincode, postcard etc.

See concepts for more details.

Goals

• Interoperability: Allows different applications to work together, even if they are using different versions of the data model.
• Data Consistency: Ensure that we process the data expected model.
• Flexibility: You can use any serialization format you want. More details here.
• Performance: A minimal overhead (encode: ~20 ns, decode: ~40 ps). More details here.

Usage

 Application 1 (DotV1) Application 2 (DotV1 and DotV2)
| |
Encode DotV1 |--------------------------------> | Decode DotV1 to DotV2
| | Modify DotV2
Decode DotV1 | <--------------------------------| Encode DotV2 back to DotV1
| |

use native_model::native_model;use serde::{Deserialize,Serialize};#[derive(Deserialize,Serialize,PartialEq,Debug)]#[native_model(id = 1, version = 1)]structDotV1(u32,u32);#[derive(Deserialize,Serialize,PartialEq,Debug)]#[native_model(id = 1, version = 2, from = DotV1)]structDotV2{name:String,x:u64,y:u64,}implFrom<DotV1>forDotV2{fnfrom(dot:DotV1) -> Self{DotV2{name:"".to_string(),x: dot.0asu64,y: dot.1asu64,}}}implFrom<DotV2>forDotV1{fnfrom(dot:DotV2) -> Self{DotV1(dot.xasu32, dot.yasu32)}}// Application 1let dot = DotV1(1,2);let bytes = native_model::encode(&dot).unwrap();// Application 1 sends bytes to Application 2.// Application 2// We are able to decode the bytes directly into a new type DotV2 (upgrade).let(mut dot, source_version) = native_model::decode::<DotV2>(bytes).unwrap();assert_eq!(dot,DotV2{
name:"".to_string(),
x:1,
y:2});
dot.name = "Dot".to_string();
dot.x = 5;// For interoperability, we encode the data with the version compatible with Application 1 (downgrade).let bytes = native_model::encode_downgrade(dot, source_version).unwrap();// Application 2 sends bytes to Application 1.// Application 1let(dot, _) = native_model::decode::<DotV1>(bytes).unwrap();assert_eq!(dot,DotV1(5,2));

• Full example here.

Serialization format

You can use default serialization formats via the feature flags, like:

[dependencies]
native_model = { version = "0.1", features = ["bincode_2"] }

Each feature flag corresponds to a specific minor version of the serialization format. In order to avoid breaking changes, the default serialization format is the oldest one.

• bincode_1_3: bincode v1.3 (default)
• bincode_2: bincode v2.0.0-rc3
• postcard_1_0: postcard v1.0
• rpm_serde_1_3: rmp-serde v1.3

Custom serialization format

Define a struct with the name you want. This struct must implement native_model::Encode and native_model::Decode traits.

Full examples:

• bincode with encode/decode
• bincode with serde

Others examples, see the default implementations:

• bincode v1.3
• bincode v2.0 (rc)
• postcard v1.0
• rmp-serde v1.3

Notice

native_model provides implementations that rely on metadata-less formats and serde. There are known issues with some serde advanced features such as:

• #[serde(flatten)]
• #[serde(skip)]
• #[serde(skip_deserializing)]
• #[serde(skip_serializing)]
• #[serde(skip_serializing_if = "path")]
• #[serde(tag = "...")]
• #[serde(untagged)]

Or types implementing similar strategies such as serde_json::Value.

The rmp-serde serialization format can optionally support them serializing structs as maps, the RmpSerdeNamed struct is provided to support this use-case.

Data model

Define your model using the macro native_model.

Attributes:

• id = u32: The unique identifier of the model.
• version = u32: The version of the model.
• with = type: The serialization format that you use for the Encode/Decode implementation. Setup here.
• from = type: Optional, the previous version of the model.
  • type: The previous version of the model that you use for the From implementation.
• try_from = (type, error): Optional, the previous version of the model with error handling.
  • type: The previous version of the model that you use for the TryFrom implementation.
  • error: The error type that you use for the TryFrom implementation.

use native_model::native_model;use serde::{Deserialize,Serialize};#[derive(Deserialize,Serialize,PartialEq,Debug)]#[native_model(id = 1, version = 1)]structDotV1(u32,u32);#[derive(Deserialize,Serialize,PartialEq,Debug)]#[native_model(id = 1, version = 2, from = DotV1)]structDotV2{name:String,x:u64,y:u64,}// Implement the conversion between versions From<DotV1> for DotV2 and From<DotV2> for DotV1.implFrom<DotV1>forDotV2{fnfrom(dot:DotV1) -> Self{DotV2{name:"".to_string(),x: dot.0asu64,y: dot.1asu64,}}}implFrom<DotV2>forDotV1{fnfrom(dot:DotV2) -> Self{DotV1(dot.xasu32, dot.yasu32)}}#[derive(Deserialize,Serialize,PartialEq,Debug)]#[native_model(id = 1, version = 3, try_from = (DotV2, anyhow::Error))]structDotV3{name:String,cord:Cord,}#[derive(Deserialize,Serialize,PartialEq,Debug)]structCord{x:u64,y:u64,}// Implement the conversion between versions From<DotV2> for DotV3 and From<DotV3> for DotV2.implTryFrom<DotV2>forDotV3{typeError = anyhow::Error;fntry_from(dot:DotV2) -> Result<Self,Self::Error>{Ok(DotV3{name: dot.name,cord:Cord{x: dot.x,y: dot.y},})}}implTryFrom<DotV3>forDotV2{typeError = anyhow::Error;fntry_from(dot:DotV3) -> Result<Self,Self::Error>{Ok(DotV2{name: dot.name,x: dot.cord.x,y: dot.cord.y,})}}

Codecs

native_model comes with several optional built-in serializer features available:

• bincode 1.3

  • This is the default codec.
  • Warning: This codec may not work with all serde-derived types.

• bincode 2.0.0-rc.3

  • Enable the bincode_2 feature and use the native_model::bincode_2::Bincode attribute to have native_db use this crate for serializing & deserializing.
  • Warning: This codec may not work with all serde-derived types.

• postcard 1.0

  • Enable the postcard_1_0 feature and use the native_model::postcard_1_0::PostCard attribute.
  • Warning: This codec may not work with all serde-derived types.

• rmp-serde 1.3

  • Enable the rmp_serde_1_3 feature and use the native_model::rmp_serde_1_3::RmpSerde attribute.

Codec example:

As example, to use rmp-serde:

1. In your project's Cargo.toml file, enable the rmp_serde_1_3 feature for the native_model dependency.
   • Be sure to check crates.io for the most recent native_model version number.

[dependencies]
serde = { version = "1.0", features = [ "derive" ] }
native_model = { version = "0.4", features = [ "rmp_serde_1_3" ] }

2. Assign the rmp_serde_1_3 codec to your struct using the with attribute:

use native_model::native_model;#[derive(Clone,Default, serde::Deserialize, serde::Serialize)]#[native_model(id = 1, version = 1, with = native_model::rmp_serde_1_3::RmpSerde)]structMyStruct{my_string:String,// etc.}

Additional reading

You may also want to check out David Koloski's Rust serialization benchmarks for help selecting the codec (i.e. bincode_1_3, rmp_serde_1_3, etc.) that's best for your project.

Status

Early development. Not ready for production.

Concepts

In order to understand how the native model works, you need to understand the following concepts.

• Identity(id): The identity is the unique identifier of the model. It is used to identify the model and prevent to decode a model into the wrong Rust type.
• Version(version) The version is the version of the model. It is used to check the compatibility between two models.
• Encode: The encode is the process of converting a model into a byte array.
• Decode: The decode is the process of converting a byte array into a model.
• Downgrade: The downgrade is the process of converting a model into a previous version of the model.
• Upgrade: The upgrade is the process of converting a model into a newer version of the model.

Under the hood, the native model is a thin wrapper around serialized data. The id and the version are twice encoded with a little_endian::U32. That represents 8 bytes, that are added at the beginning of the data.

+------------------+------------------+------------------------------------+
| ID (4 bytes) | Version (4 bytes)| Data (indeterminate-length bytes) |
+------------------+------------------+------------------------------------+

Full example here.

Performance

Native model has been designed to have a minimal and constant overhead. That means that the overhead is the same whatever the size of the data. Under the hood we use the zerocopy crate to avoid unnecessary copies.

👉 To know the total time of the encode/decode, you need to add the time of your serialization format.

Resume:

• Encode: ~20 ns
• Decode: ~40 ps

data size	encode time (ns)	decode time (ps)
1 B	19.769 ns - 20.154 ns	40.526 ps - 40.617 ps
1 KiB	19.597 ns - 19.971 ns	40.534 ps - 40.633 ps
1 MiB	19.662 ns - 19.910 ns	40.508 ps - 40.632 ps
10 MiB	19.591 ns - 19.980 ns	40.504 ps - 40.605 ps
100 MiB	19.669 ns - 19.867 ns	40.520 ps - 40.644 ps

Benchmark of the native model overhead here.