preprocessing.rs

Overview

This file implements the preprocessing logic for optimistic shard state updates in a multi-threaded blockchain system. It primarily handles the integration and preparation of shard state, cross-thread references, and message queues before the main block processing occurs. The preprocessing includes merging thread tables, importing migrating accounts with their inboxes, settling accounts, and preparing message queues with settled and high-priority messages such as slashing and epoch-related messages.

The key output of this file is a PreprocessingResult struct that contains the updated shard state, the thread table, and collections of redirected and settled messages.

Structs and Types

PreprocessingResult

pub struct PreprocessingResult {
    pub state: State,
    pub threads_table: ThreadsTable,
    pub redirected_messages: HashMap<AccountRouting, Vec<(MessageIdentifier, Arc<WrappedMessage>)>>,
    pub settled_messages: HashMap<AccountAddress, Vec<(MessageIdentifier, Arc<WrappedMessage>)>>,
}

• Purpose: Holds the result of running the preprocessing step on the shard state.

• Fields:

  • state: The updated optimistic shard state after preprocessing.

  • threads_table: The threads table associated with the current state.

  • redirected_messages: Messages that have been redirected to different routes, grouped by AccountRouting.

  • settled_messages: Messages that have been settled (delivered or finalized), grouped by AccountAddress.

Functions

preprocess

pub fn preprocess<'a, I, TRepo>(
    parent_block_state: State,
    refs: I,
    descendant_thread_identifier: &ThreadIdentifier,
    repository: &TRepo,
    slashing_messages: Vec<Arc<WrappedMessage>>,
    epoch_block_keeper_data: Vec<BlockKeeperData>,
    message_db: MessageDurableStorage,
    metrics: Option<BlockProductionMetrics>,
) -> anyhow::Result<PreprocessingResult>
where
    I: std::iter::Iterator<Item = &'a CrossThreadRefData>,
    CrossThreadRefData: 'a,
    TRepo: CrossThreadRefDataRead,

• Purpose: Performs the preprocessing step on the given parent block state using cross-thread references and other inputs.

• Parameters:

  • parent_block_state: The optimistic shard state from the parent block.

  • refs: Iterator over cross-thread reference data used for merging referenced states.

  • descendant_thread_identifier: The thread ID of the descendant thread to preprocess.

  • repository: Repository interface to fetch cross-thread reference data.

  • slashing_messages: A vector of slashing-related messages to be included as high-priority.

  • epoch_block_keeper_data: Data related to epoch block keeper, used to create epoch touch messages.

  • message_db: Durable storage interface for messages.

  • metrics: Optional block production metrics collector.

• Returns: PreprocessingResult containing the updated state and message sets.

• Key Steps:

  1. Clone the thread table from the parent state.

  2. Update thread references in the state to ensure the parent reference is included.

  3. Collect all added cross-thread references from the provided iterator.

  4. Sort and merge the referenced blocks.

  5. Merge thread tables from referenced blocks.

  6. Crop the shard state for the descendant thread.

  7. Import migrating accounts and their inboxes from referenced blocks.

  8. Settle messages that belong to the descendant thread.

  9. Build the message queue state with settled messages.

  10. Convert slashing and epoch messages into high-priority messages.

  11. Build the message queue state for high-priority messages.

• Usage Example:

let preprocessing_result = preprocess(
    parent_state,
    cross_thread_refs_iter,
    &descendant_thread_id,
    &repository,
    slashing_msgs,
    epoch_data,
    message_storage,
    Some(metrics),
)?;

convert_slashing_messages

pub fn convert_slashing_messages(
    slashing_messages: Vec<Arc<WrappedMessage>>,
) -> anyhow::Result<HashMap<AccountAddress, Vec<(MessageIdentifier, Arc<WrappedMessage>)>>>

• Purpose: Converts a vector of slashing messages into a map keyed by destination account addresses.

• Parameters:

  • slashing_messages: Vector of wrapped slashing messages.

• Returns: A map from AccountAddress to a vector of (MessageIdentifier, WrappedMessage) tuples.

• Implementation Details:

  • Extracts the destination address from each message's internal header.

  • Groups messages by their destination account address.

convert_epoch_messages

pub fn convert_epoch_messages(
    high_priority_map: &mut HashMap<AccountAddress, Vec<(MessageIdentifier, Arc<WrappedMessage>)>>,
    epoch_message: Vec<BlockKeeperData>,
) -> anyhow::Result<()>

• Purpose: Converts epoch block keeper data into epoch touch messages and adds them to the provided high-priority message map.

• Parameters:

  • high_priority_map: Mutable reference to the map of high-priority messages grouped by account address.

  • epoch_message: Vector of epoch block keeper data.

• Returns: Ok(()) if successful, otherwise an error.

• Implementation Details:

  • Obtains the current UNIX timestamp.

  • Creates an epoch touch message for each BlockKeeperData.

  • Wraps each message and inserts it into the map keyed by destination account address.

import_migrating_accounts_with_their_inboxes

fn import_migrating_accounts_with_their_inboxes(
    all_referenced_blocks: &[CrossThreadRefData],
    in_table: &ThreadsTable,
    descendant_thread_identifier: &ThreadIdentifier,
    mut state: State,
    message_db: MessageDurableStorage,
    metrics: Option<BlockProductionMetrics>,
) -> anyhow::Result<State>

• Purpose: Imports migrating accounts and their inboxes from all referenced cross-thread blocks into the local shard state.

• Parameters:

  • all_referenced_blocks: Slice of referenced cross-thread block data.

  • in_table: The current threads table.

  • descendant_thread_identifier: The thread ID for which preprocessing is done.

  • state: The current optimistic shard state.

  • message_db: Durable message storage.

  • metrics: Optional metrics collector.

• Returns: Updated optimistic shard state including imported accounts and inboxes.

• Algorithm:

  • Iterates over outbound accounts in referenced blocks.

  • Checks if accounts are relevant to the current thread using the thread table.

  • Collects migrated accounts and their inboxes.

  • Reports outbound accounts count via metrics if available.

  • Calls settle_accounts to update the underlying shard state with migrated and removed accounts.

  • Updates the message queue state with added inboxes.

• Usage Context: Called within preprocess to merge accounts from cross-thread references.

settle_accounts

fn settle_accounts(
    shard_state: &mut OptimisticShardState,
    migrated_accounts: Vec<WrappedAccount>,
    removed_accounts: Vec<AccountAddress>,
) -> anyhow::Result<()>

• Purpose: Updates the shard state by adding migrated accounts and removing accounts that no longer belong.

• Parameters:

  • shard_state: Mutable reference to the optimistic shard state.

  • migrated_accounts: List of accounts to be added or updated.

  • removed_accounts: List of account addresses to be removed.

• Returns: Ok(()) if successful, otherwise an error.

• Implementation Details:

  • Converts the optimistic shard state to a mutable ShardStateUnsplit.

  • Reads existing accounts from the shard state.

  • Inserts migrated accounts and removes specified accounts.

  • Writes updated accounts back into the shard state.

• Notes: Uses interior mutability via Arc::make_mut on the shard state wrapper.

Important Implementation Details and Algorithms

• Cross-Thread Reference Handling:
  The preprocess function collects and merges cross-thread references by fetching referenced blocks and sorting them by sequence number and block identifier to maintain deterministic ordering.

• Thread Table Merging:
  Thread tables from referenced blocks are merged into the current state's thread table to synchronize thread routing information.

• Cropping:
  The state is cropped to the descendant thread using the crop function (defined elsewhere), which likely prunes state related to irrelevant threads or messages.

• Account Migration:
  Migrating accounts are imported from referenced blocks, filtered by thread table membership, and settled into the state with inboxes properly updated.

• Message Queue Construction:
  Message queues for settled messages and high-priority messages (including slashing and epoch messages) are built via ThreadMessageQueueState::build_next() builder pattern, ensuring the state is consistent and ready for block processing.

• High-Priority Messages:
  Slashing and epoch touch messages are converted and inserted into the high-priority message queue to ensure timely processing during block production.

Interactions with Other Components

• OptimisticState and OptimisticShardState:
  The file interacts heavily with these types from the repository module to manipulate shard states optimistically.

• CrossThreadRefData and CrossThreadRefDataRead:
  Interfaces used to read and process cross-thread referenced data required for merging states from other threads.

• ThreadMessageQueueState:
  Used to build and update message queues for both settled and high-priority messages.

• MessageDurableStorage:
  Provides persistent storage access for messages during preprocessing.

• BlockKeeperData and Epoch Messages:
  Epoch-related data is transformed into epoch touch messages and queued as high-priority messages.

• WrappedMessage and MessageIdentifier:
  Abstractions for blockchain message handling and identification.

• AccountRouting, AccountAddress, and ThreadsTable:
  Used for routing and thread membership checks during account migration and message settlement.

• Metrics Reporting:
  Optional integration with BlockProductionMetrics to report outbound accounts during preprocessing.

Visual Diagram

flowchart TD
A[preprocess] --> B[Update Thread References]
B --> C[Fetch Cross-Thread References]
C --> D[Sort & Merge Referenced Blocks]
D --> E[Merge Threads Table]
E --> F[Crop Shard State]
F --> G[import_migrating_accounts_with_their_inboxes]
G --> H[Settle Accounts]
H --> I[Build Settled Messages Queue]
I --> J[convert_slashing_messages]
J --> K[convert_epoch_messages]
K --> L[Build High-Priority Messages Queue]
L --> M[Return PreprocessingResult]
G -.-> F
H -.-> G
I -.-> H
J -.-> I
K -.-> J
L -.-> K

• The diagram shows the main workflow inside preprocess and the calls to helper functions for account migration, settlement, and message queue building.

• Dashed arrows represent internal data flow dependencies between stages.

References to Related Topics

• For details on optimistic shard state management and internal state mutation, see OptimisticShardState.

• For how CrossThreadRefData and cross-thread references work for state synchronization, see CrossThreadRefData Handling.

• The message queue building and management are described in ThreadMessageQueueState.

• Account routing and threads table concepts are elaborated under Account Routing and Threads Table Management.

• For transaction and message identification, see MessageIdentifier and WrappedMessage.

• Metrics integration for block production is covered in BlockProductionMetrics.

• Epoch-related block keeper data and touch message creation is described in Epoch Block Keeper Data.