optimistic_state.rs

Overview

This file defines the core structures and logic for managing an optimistic state representation of a blockchain shard's state in an asynchronous and concurrent environment. The optimistic state enables independent processing and verification of blocks and messages before final confirmation on the main chain, facilitating parallelism and efficiency in blockchain state management.

The file includes a trait OptimisticState defining the interface for optimistic state implementations, and a concrete implementation OptimisticStateImpl that encapsulates the shard's state, message queues, thread references, and related metadata. It supports serialization/deserialization, application of new blocks, state cropping based on thread partitions, and management of cross-thread messages and slashing messages.

Key Components

Trait: OptimisticState

Defines the behavior expected from any optimistic state implementation.

Associated Types

• Cell: Underlying serialized cell type representing state.

• Message: Type representing messages processed by the state (must implement Message).

• ShardState: The type representing the shard's blockchain state.

Required Methods

• get_share_stare_refs(&self) -> HashMap<ThreadIdentifier, BlockIdentifier>
  Returns a map of thread IDs to the latest block IDs that are referenced by the state.

• get_block_seq_no(&self) -> &BlockSeqNo
  Returns the sequence number of the last applied block.

• get_block_id(&self) -> &BlockIdentifier
  Returns the identifier of the last applied block.

• get_shard_state(&self) -> Self::ShardState
  Returns the current shard state.

• get_shard_state_as_cell(&self) -> Self::Cell
  Returns the shard state represented as a serialized cell.

• get_block_info(&self) -> &BlockInfo
  Returns metadata information about the block.

• serialize_into_buf(self) -> anyhow::Result<Vec<u8>>
  Serializes the entire optimistic state into a binary buffer.

• apply_block(&mut self, block_candidate: &AckiNackiBlock, shared_services: &SharedServices, block_state_repo: BlockStateRepository, nack_set_cache: Arc<Mutex<FixedSizeHashSet<UInt256>>>, accounts_repo: AccountsRepository, message_db: MessageDurableStorage) -> anyhow::Result<(CrossThreadRefData, HashMap<AccountAddress, Vec<(MessageIdentifier, Arc<WrappedMessage>)>>)>
  Applies a new block to the optimistic state, updating internal state and message queues accordingly.

• get_thread_id(&self) -> &ThreadIdentifier
  Returns the thread ID this state belongs to.

• get_produced_threads_table(&self) -> &ThreadsTable
  Returns the threads table representing thread assignments for accounts.

• set_produced_threads_table(&mut self, table: ThreadsTable)
  Sets a new threads table.

• crop(&mut self, thread_identifier: &ThreadIdentifier, threads_table: &ThreadsTable, message_db: MessageDurableStorage) -> anyhow::Result<()>
  Crops the state by removing data not relevant to the specified thread and threads table.

• get_thread_for_account(&self, account_routing: &AccountRouting) -> anyhow::Result<ThreadIdentifier>
  Returns the thread ID for a given account routing.

• does_routing_belong_to_the_state(&self, account_routing: &AccountRouting) -> bool
  Checks if a given account routing belongs to the current thread of this state.

• get_internal_message_queue_length(&self) -> usize
  Returns the length of the internal message queue.

• does_state_has_messages_to_other_threads(&mut self) -> anyhow::Result<bool>
  Checks if the state contains messages destined for other threads.

• add_slashing_messages(&mut self, slashing_messages: Vec<Arc<Self::Message>>) -> anyhow::Result<()>
  Adds slashing messages to the state's outgoing message queue.

• get_thread_refs(&self) -> &ThreadReferencesState
  Returns the current thread references maintained by the state.

Struct: CrossThreadMessageData

Represents metadata about messages that cross thread boundaries.

Fields

• block_id: BlockIdentifier — Identifier of the block that produced the message.

• src_account_routing: AccountRouting — Source account routing used for message cropping and splitting.

• dest_account_routing: AccountRouting — Destination account routing, used for tracking the message.

Struct: OptimisticStateImpl

Concrete implementation of the OptimisticState trait, representing the optimistic state of a shard.

Fields

• block_seq_no: BlockSeqNo — Sequence number of the last applied block.

• block_id: BlockIdentifier — Identifier of the last applied block.

• shard_state: OptimisticShardState — The current shard state.

• messages: ThreadMessageQueueState — Queue of normal priority messages.

• high_priority_messages: ThreadMessageQueueState — Queue of high priority messages.

• block_info: BlockInfo — Metadata about the block.

• threads_table: ThreadsTable — Table mapping accounts to threads.

• thread_id: ThreadIdentifier — Thread identifier for this state.

• thread_refs_state: ThreadReferencesState — References to other threads' block states.

• cropped: Option<(ThreadIdentifier, ThreadsTable)> — Optional data indicating if the state was cropped and by which thread/table.

• changed_accounts: HashMap<AccountAddress, BlockSeqNo> — Tracks accounts changed in current state (not serialized).

• cached_accounts: HashMap<AccountAddress, (BlockSeqNo, Cell)> — Cached account data (not serialized).

• accounts_number: u64 (optional, feature-gated) — Count of accounts monitored.

Important Methods

• make_an_independent_copy(&mut self)
  Creates an independent copy of the shard state to avoid shared references.

• messages(&self) -> &ThreadMessageQueueState
  Returns a reference to the message queue state.

• deserialize_from_buf(data: &[u8]) -> anyhow::Result<Self>
  Deserializes an OptimisticStateImpl from a byte buffer.

• zero() -> Self
  Returns a zeroed/default instance with empty state.

• set_shard_state(&mut self, new_state: <Self as OptimisticState>::ShardState)
  Sets the shard state from an external shard state instance.

• load_changed_accounts(&self, state: &mut ShardStateUnsplit, block: &tvm_block::Block, accounts_repo: AccountsRepository) -> anyhow::Result<HashSet<AccountAddress>>
  Loads accounts that have changed in the given block and updates the shard state accordingly.

• save_to_file(self, path: &Path) -> anyhow::Result<()>
  Serializes and saves the optimistic state to a specified file in a safe manner (using temporary file and atomic rename).

• load_from_file(path: &Path) -> anyhow::Result<Self>
  Loads an optimistic state from a serialized file.

• apply_block(...) -> anyhow::Result<(CrossThreadRefData, HashMap<AccountAddress, Vec<(MessageIdentifier, Arc<WrappedMessage>)>>)>
  Applies a new block to the optimistic state, performing necessary state updates, message processing, and postprocessing steps.

• crop(&mut self, thread_identifier: &ThreadIdentifier, threads_table: &ThreadsTable, message_db: MessageDurableStorage) -> anyhow::Result<()>
  Crops the state to only include data relevant to the specified thread and threads table, removing irrelevant accounts and messages.

• get_thread_for_account(&self, account_routing: &AccountRouting) -> anyhow::Result<ThreadIdentifier>
  Finds the thread ID that corresponds to a given account routing, using the threads table.

• does_routing_belong_to_the_state(&self, account_routing: &AccountRouting) -> bool
  Checks if the given routing belongs to the current thread.

• does_state_has_messages_to_other_threads(&mut self) -> anyhow::Result<bool>
  Determines if the current state contains messages destined for other threads by inspecting outgoing message queues.

• add_slashing_messages(&mut self, slashing_messages: Vec<Arc<Self::Message>>) -> anyhow::Result<()>
  Adds slashing messages to the outgoing message queue in the shard state.

• get_thread_refs(&self) -> &ThreadReferencesState
  Returns the current thread reference state.

Debug and Default Implementations

• Custom Debug implementation shows key fields.

• Default implementation returns a zeroed state.

Struct: TrimmedOptimisticStateImpl

A trimmed variant of OptimisticStateImpl used for serialization without the full shard state, which is stored separately as a Cell.

Fields

• Subset of OptimisticStateImpl fields excluding the heavy shard_state object.

• Used to reduce serialized size and separate large state cell serialization.

Conversion

• Implements From to extract trimmed data.

• state_from_trimmed(trimmed: TrimmedOptimisticStateImpl, state_cell: Cell) -> OptimisticStateImpl reconstructs full state using the trimmed data and a shard state cell.

Important Implementation Details and Algorithms

• State Application (apply_block):
  This method performs several steps when applying a block candidate:

  1. Verifies the parent block matches the current state.

  2. Processes "nack" (negative acknowledgements) messages to produce slashing messages for misbehaving nodes.

  3. Preprocesses the block with external services and references to other threads' states.

  4. Loads changed accounts from the block into the shard state, ensuring consistency of external accounts.

  5. Applies the block's state update to the current shard state.

  6. Collects and processes internal messages produced or consumed by the block.

  7. Runs postprocessing that may modify the optimistic state and cross-thread references.

  8. Updates the cache of known nacks to avoid duplicates.

  9. Tracks performance metrics for block application.

• Cropping (crop):
  The cropping operation reduces the state to only include accounts and messages relevant to a specific thread and threads table. This is critical for maintaining efficient and thread-specific optimistic states in multi-threaded blockchain processing. The method:

  • Filters the shard state based on the threads table.

  • Removes accounts and messages not matching the thread.

  • Rebuilds the message queue state with updated account lists.

• Message Queues and Cross-Thread References:
  The optimistic state maintains separate message queues for normal and high-priority messages. It also tracks references to other threads' block states to manage cross-thread message passing and synchronization.

• Serialization and Storage:
  The state is serialized into two parts: the metadata (a trimmed optimistic state) and the shard state as a serialized cell. This separation optimizes storage and load times. File operations use temporary files and atomic renames to ensure consistency.

• Slashing Messages:
  Slashing messages are generated during block application from nack data and are added to the outgoing message queue with special handling to prioritize processing.

Interactions with Other Modules

• tvm_block: Uses types such as Block, ShardStateUnsplit, and StateUpdate for block data and state updates.

• message: Manages message representations, envelopes, and message identifiers.

• multithreading: Interacts with thread references and shard state cropping utilities.

• block_keeper_system: Generates slashing messages for misbehaving nodes.

• repository: Accesses cross-thread reference data and account repositories.

• storage: Utilizes durable message storage for queue management.

• types: Uses blockchain-specific types like BlockIdentifier, ThreadIdentifier, AccountRouting, and ThreadsTable.

• helper: Uses utility functions for file operations.

• tracing: Uses instrumentation macros for logging and tracing execution.

Usage Examples

Applying a Block to an Optimistic State

let mut optimistic_state: OptimisticStateImpl = ...; // existing state
let block_candidate: AckiNackiBlock = ...; // new block to apply
let shared_services: SharedServices = ...;
let block_state_repo: BlockStateRepository = ...;
let nack_set_cache = Arc::new(Mutex::new(FixedSizeHashSet::new()));
let accounts_repo: AccountsRepository = ...;
let message_db: MessageDurableStorage = ...;

let (cross_thread_ref_data, added_messages) = optimistic_state
    .apply_block(&block_candidate, &shared_services, block_state_repo, nack_set_cache, accounts_repo, message_db)?;

Cropping the State to a Specific Thread

let thread_id: ThreadIdentifier = ...;
let threads_table: ThreadsTable = ...;
let message_db: MessageDurableStorage = ...;

optimistic_state.crop(&thread_id, &threads_table, message_db)?;

Serializing and Saving the State

let file_path = Path::new("state_file.dat");
optimistic_state.save_to_file(file_path)?;

Loading State from a File

let loaded_state = OptimisticStateImpl::load_from_file(file_path)?;

Data Structure Diagram

classDiagram
class OptimisticStateImpl {
-block_seq_no: BlockSeqNo
-block_id: BlockIdentifier
-shard_state: OptimisticShardState
-messages: ThreadMessageQueueState
-high_priority_messages: ThreadMessageQueueState
-block_info: BlockInfo
-threads_table: ThreadsTable
-thread_id: ThreadIdentifier
-thread_refs_state: ThreadReferencesState
-cropped: Option
-changed_accounts: HashMap
-cached_accounts: HashMap
+make_an_independent_copy()
+deserialize_from_buf()
+zero()
+set_shard_state()
+save_to_file()
+load_from_file()
+apply_block()
+crop()
+add_slashing_messages()
+get_thread_refs()
}
class OptimisticState {
<<interface>>
+get_share_stare_refs()
+get_block_seq_no()
+get_block_id()
+get_shard_state()
+get_shard_state_as_cell()
+get_block_info()
+serialize_into_buf()
+apply_block()
+get_thread_id()
+get_produced_threads_table()
+set_produced_threads_table()
+crop()
+get_thread_for_account()
+does_routing_belong_to_the_state()
+get_internal_message_queue_length()
+does_state_has_messages_to_other_threads()
+add_slashing_messages()
+get_thread_refs()
}
OptimisticStateImpl ..|> OptimisticState
class CrossThreadMessageData {
+block_id: BlockIdentifier
+src_account_routing: AccountRouting
+dest_account_routing: AccountRouting
}
class TrimmedOptimisticStateImpl {
+block_seq_no: BlockSeqNo
+block_id: BlockIdentifier
+messages: ThreadMessageQueueState
+high_priority_messages: ThreadMessageQueueState
+block_info: BlockInfo
+threads_table: ThreadsTable
+thread_id: ThreadIdentifier
+thread_refs_state: ThreadReferencesState
+cropped: Option
}
OptimisticStateImpl --> "1" OptimisticShardState
OptimisticStateImpl --> "1" ThreadMessageQueueState : messages
OptimisticStateImpl --> "1" ThreadMessageQueueState : high_priority_messages
OptimisticStateImpl --> "1" BlockInfo
OptimisticStateImpl --> "1" ThreadsTable
OptimisticStateImpl --> "1" ThreadReferencesState

Additional Notes

• The apply_block method heavily relies on external services such as SharedServices, BlockStateRepository, and AccountsRepository for block processing and cross-thread reference data management.

• The cropping mechanism is essential for managing state size and ensuring that the optimistic state remains thread-specific, which is crucial for correct parallel processing of shard states.

• The file uses Rust features such as traits, generics, and advanced serialization libraries (serde, bincode), reflecting careful design for modularity and performance.

• The optimistic state manages both normal and high-priority messages separately, allowing prioritization in message handling.

• Error handling is done via the anyhow crate, providing context-rich errors.

• Thread safety is ensured using synchronization primitives like parking_lot::Mutex and Arc.

For further details on related types and concepts, see Block Identifiers and Sequence Numbers, Thread and Shard Management, Message Queues, and Shard State Updates.