serialize.rs

Overview

This file provides a custom binary serialization and deserialization framework tailored for the Chitchat system. The primary goals of this serialization format are to:

• Efficiently serialize various data types including primitives, IP addresses, socket addresses, and domain-specific types such as ChitchatId and Heartbeat.

• Support truncation of serialized payloads to fit within a configurable maximum transmission unit (MTU).

• Enable compression of serialized data blocks using zstd compression, while falling back to uncompressed blocks if compression is not beneficial.

• Facilitate streaming serialization and deserialization with block-based compression.

The file defines two core traits, Serializable and Deserializable, for encoding and decoding types, respectively. It also implements these traits for a range of standard and custom types. A key component is the CompressedStreamWriter, which manages block-based serialization with optional compression.

Traits

Serializable

Trait defining the interface for types that can be serialized into a byte buffer.

• Methods:

  • fn serialize(&self, buf: &mut Vec<u8>): Serializes the instance into the provided buffer.

  • fn serialize_to_vec(&self) -> Vec<u8>: Convenience method that serializes to a new Vec<u8>.

  • fn serialized_len(&self) -> usize: Returns the length in bytes of the serialized representation.

Deserializable

Trait defining the interface for types that can be deserialized from a byte slice.

• Associated type constraint: Sized

• Methods:

  • fn deserialize(buf: &mut &[u8]) -> anyhow::Result<Self>: Attempts to read an instance from the start of buf, advancing the slice.

Implementations of Serializable and Deserializable

The file provides implementations for primitive types, standard library types, and Chitchat-specific types:

• u8, u16, u32, u64: Serialized in little-endian byte order.

• bool: Serialized as a single byte, 0 for false and 1 for true.

• IpAddr (IPv4 and IPv6): Serialized with a preceding version byte (4 or 6), followed by the raw octets.

• SocketAddr: Serialized as IP address followed by port number.

• String and str: Serialized as a u16 length prefix followed by UTF-8 bytes.

• Fixed-size [u8; N] arrays: Serialized as raw bytes.

• ChitchatId: Serializes its node_id (string), generation_id (u64), and gossip_advertise_addr (SocketAddr).

• Heartbeat: Wraps a u64 and serializes that value.

Example: Serializing an IpAddr

let ip = IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1));
let mut buf = Vec::new();
ip.serialize(&mut buf);
// buf will contain: [4, 127, 0, 0, 1] 

Example: Deserializing a SocketAddr

let mut buf: &[u8] = /* some byte slice */;
let socket_addr = SocketAddr::deserialize(&mut buf)?;

IpVersion Enum

Internal enumeration to represent IP address version (IPv4 or IPv6) during serialization.

• V4 = 4u8

• V6 = 6u8

Implements TryFrom<u8> for safe conversion from a byte to IpVersion.

BlockType Enum

Enumerates the types of blocks used in the compression stream:

• NoMoreBlocks: Indicates the end of the stream.

• Compressed: A compressed block.

• Uncompressed: An uncompressed block.

Each variant is represented as a single byte (0, 1, or 2) and implements serialization/deserialization.

CompressedStreamWriter

Purpose

Manages the serialization of a sequence of Serializable items into a compressed stream with the following features:

• Splits data into blocks of a configurable maximum size (block_threshold).

• Compresses each block using zstd.

• If compression results in a larger block, stores the block uncompressed.

• Tags blocks with their type to allow proper decoding.

• Provides an upper bound estimate on the serialized length after appending an item, enabling MTU enforcement.

Fields

• output: Vec<u8>: The final output buffer accumulating serialized blocks.

• uncompressed_block: Vec<u8>: Temporary buffer collecting serialized items before compression.

• compressed_block: Vec<u8>: Temporary buffer for compressed data.

• block_threshold: usize: Maximum uncompressed block size before flushing.

Methods

with_block_threshold(block_threshold: u16) -> Self

Creates a new CompressedStreamWriter with the specified block size threshold.

serialized_len_upperbound_after<S: Serializable + ?Sized>(&self, item: &S) -> usize

Estimates an upper bound on the total serialized length after appending item.

• Takes into account whether the current block will overflow and need to be flushed.

• Includes metadata overhead for block headers and terminal tag.

append<S: Serializable + ?Sized>(&mut self, item: &S)

Appends a serializable item to the stream.

• Serializes the item into the uncompressed_block buffer.

• Flushes full blocks (over threshold) by compressing or storing uncompressed.

flush_block(&mut self)

Flushes the first block_threshold bytes from uncompressed_block.

• Attempts to compress using zstd.

• If compression fails or is larger than uncompressed size, stores uncompressed.

• Writes block type, length, and bytes to the output buffer.

• Removes flushed bytes from uncompressed_block.

finish(self) -> Vec<u8>

Finalizes the stream:

• Flushes remaining buffered data.

• Writes the NoMoreBlocks terminator.

• Returns the complete serialized output.

Function: deserialize_stream

fn deserialize_stream<D: Deserializable>(buf: &mut &[u8]) -> anyhow::Result<Vec<D>>

Deserializes a block-compressed stream from a byte buffer.

• Reads blocks one by one by their BlockType.

• Decompresses compressed blocks or copies uncompressed blocks into an internal buffer.

• Stops at NoMoreBlocks.

• Deserializes individual items of type D from the decompressed data.

• Returns a vector of deserialized items.

Implementation Details and Algorithms

• Primitive types are serialized in little-endian to ensure cross-platform consistency.

• IP addresses are serialized with a version byte to distinguish IPv4 and IPv6.

• Strings are length-prefixed with a u16 allowing up to 65535 bytes.

• Serialization of arrays and slices uses raw byte copying.

• The CompressedStreamWriter uses zstd compression at the default level.

• Compression is adaptive: if compressed data is larger than uncompressed, the block is stored uncompressed to avoid inefficiency.

• Blocks are prefixed with a 1-byte type and a 2-byte length to allow correct parsing by the deserializer.

• The serialization format supports truncation and MTU enforcement via block boundaries and length calculations.

Interaction with Other Parts of the System

• Uses domain-specific types ChitchatId and Heartbeat from the crate root which have custom serialization logic here.

• Depends on external crates:

  • anyhow for error handling.

  • bytes::Buf trait for buffer management.

  • zstd for compression/decompression.

• Works with network-related types such as IpAddr and SocketAddr to serialize network addresses used in the Chitchat protocol.

• Supports streaming serialization and deserialization, enabling efficient network communication of Chitchat messages.

Visual Diagram

classDiagram
class Serializable {
<<trait>>
+serialize(buf: &mut Vec<u8>)
+serialize_to_vec() Vec<u8>
+serialized_len() usize
}
class Deserializable {
<<trait>>
+deserialize(buf: &mut &[u8]) Result<Self>
}
class CompressedStreamWriter {
-output: Vec<u8>
-uncompressed_block: Vec<u8>
-compressed_block: Vec<u8>
-block_threshold: usize
+with_block_threshold(u16) CompressedStreamWriter
+serialized_len_upperbound_after(item: &Serializable) usize
+append(item: &Serializable)
-flush_block()
+finish() Vec<u8>
}
class BlockType {
<<enum>>
+NoMoreBlocks
+Compressed
+Uncompressed
+serialize(buf: &mut Vec<u8>)
+deserialize(buf: &mut &[u8]) Result<Self>
}
Serializable <|.. u8
Serializable <|.. u16
Serializable <|.. u32
Serializable <|.. u64
Serializable <|.. bool
Serializable <|.. IpAddr
Serializable <|.. SocketAddr
Serializable <|.. String
Serializable <|.. str
Serializable <|.. [u8; N]
Serializable <|.. ChitchatId
Serializable <|.. Heartbeat
Deserializable <|.. u8
Deserializable <|.. u16
Deserializable <|.. u32
Deserializable <|.. u64
Deserializable <|.. bool
Deserializable <|.. IpAddr
Deserializable <|.. SocketAddr
Deserializable <|.. String
Deserializable <|.. [u8; N]
Deserializable <|.. ChitchatId
Deserializable <|.. Heartbeat
CompressedStreamWriter --> Serializable : uses
deserialize_stream ..> Deserializable : uses
CompressedStreamWriter --> BlockType : manages blocks

Usage Examples

Serializing and Deserializing a ChitchatId

let id = ChitchatId::new("node-123".to_string(), 42, "127.0.0.1:8080".parse().unwrap());
let mut buffer = Vec::new();
id.serialize(&mut buffer);

let mut slice = &buffer[..];
let deserialized_id = ChitchatId::deserialize(&mut slice)?;
assert_eq!(id, deserialized_id);

Using CompressedStreamWriter to Serialize Multiple Items

let mut writer = CompressedStreamWriter::with_block_threshold(1024);

for message in messages {
    writer.append(&message);
}

let compressed_bytes = writer.finish();

// Send `compressed_bytes` over network or store.

Deserializing a Compressed Stream

let mut slice = &compressed_bytes[..];
let items: Vec<String> = deserialize_stream(&mut slice)?;

Testing

The file includes comprehensive unit tests and property-based tests to verify correctness of:

• Serialization and deserialization of all supported types.

• Block compression and decompression logic.

• Edge cases such as empty strings and random data.

• Length upper bound estimation accuracy.

These tests validate the robustness and correctness of the serialization framework.

References

• For error handling patterns see anyhow crate.

• For buffer utilities see bytes::Buf.

• For compression algorithms see zstd.

• For network address types see std::net::IpAddr, std::net::SocketAddr.

• For domain-specific types see ChitchatId, Heartbeat.