shard_state_operations.rs

Overview

This file provides functionality to manipulate and filter shard states within a distributed system that partitions data and messages across multiple threads. Specifically, it offers a method to crop the shard state based on a given thread identifier and a threads routing table. Cropping here means filtering out accounts and outgoing messages that do not belong to the specified thread, according to routing rules defined by the ThreadsTable. This ensures that each thread processes only relevant data, and cross-thread messages are separated for appropriate handling.

The primary operation is performed by the function crop_shard_state_based_on_threads_table, which:

• Filters outgoing messages, removing those destined for accounts not matching the thread.

• Filters accounts, removing or redirecting those not matching the thread.

• Uses routing tables and account routing to determine thread membership.

• Provides callbacks for account handling and buffers for removed accounts.

• Returns a new filtered shard state.

This operation is crucial for maintaining data integrity and efficiency in multi-threaded shard processing scenarios.

Detailed Explanation

Function: crop_shard_state_based_on_threads_table

pub(crate) fn crop_shard_state_based_on_threads_table<F>(
    shard_state: Arc<ShardStateUnsplit>,
    threads_table: &ThreadsTable,
    thread_id: ThreadIdentifier,
    _block_id: BlockIdentifier,
    optimization_skip_shard_accounts_crop: bool,
    removed_accounts_buffer: &mut Vec<AccountAddress>,
    mut on_account_callback: F,
) -> anyhow::Result<Arc<ShardStateUnsplit>>
where
    F: FnMut(&AccountId),

Purpose

Filters the given shard state to only include accounts and outgoing messages relevant to the specified thread, according to the provided threads routing table.

Parameters

• shard_state: Arc<ShardStateUnsplit>
  The original shard state containing accounts and outgoing messages to be filtered.

• threads_table: &ThreadsTable
  A routing table used to determine if an account or message belongs to a specific thread.

• thread_id: ThreadIdentifier
  The identifier of the thread for which the shard state should be cropped.

• _block_id: BlockIdentifier
  A block identifier parameter, currently unused (marked as TODO for removal).

• optimization_skip_shard_accounts_crop: bool
  Flag to skip the cropping of accounts for optimization purposes.

• removed_accounts_buffer: &mut Vec<AccountAddress>
  Mutable buffer to collect addresses of accounts removed during cropping.

• on_account_callback: F
  A callback function invoked for each account processed during cropping. It receives a reference to AccountId.

Returns

• anyhow::Result<Arc<ShardStateUnsplit>>
  Returns a new ShardStateUnsplit wrapped in an Arc that contains only the filtered accounts and messages corresponding to the given thread. On failure, returns an error.

Usage Example

let mut removed_accounts = Vec::new();
let filtered_shard_state = crop_shard_state_based_on_threads_table(
    original_shard_state,
    &threads_table,
    thread_id,
    block_id,
    false,
    &mut removed_accounts,
    |account_id| {
        println!("Processed account: {:?}", account_id);
    },
)?;

Implementation Details

• Reading Accounts and Messages:
  The function clones the given shard_state and reads both accounts and outgoing message queues from it.

• Filtering Outgoing Messages:
  It iterates through all outgoing messages, extracting the message destination routing information from the message header. If the destination account routing does not match the thread as per the threads_table, the message key is recorded for removal from the outgoing queue.

• Removing Messages:
  Messages identified for removal are deleted from the outgoing queue and the updated queue is written back to the shard state.

• Account Filtering (Conditional):
  If the optimization_skip_shard_accounts_crop flag is false, accounts are iterated over and their routing is computed via get_account_routing_for_account. Accounts not matching the thread are either removed or replaced with redirect accounts based on conditions derived from the routing mask.

• Callbacks and Buffers:
  The on_account_callback is called for each account processed. Removed accounts' addresses are collected in the removed_accounts_buffer for further processing outside this function.

• Error Handling:
  Throughout, errors in reading, iterating, removing, or writing accounts and messages are propagated wrapped with contextual messages using anyhow.

• Thread Routing Matching:
  The core logic for determining whether an account or message belongs to a thread relies on threads_table.is_match, which compares the routing of an account or message destination with the given thread_id.

Key Types and Concepts

• ShardStateUnsplit
  Represents the state of a shard, containing accounts and queues of outgoing messages.

• ThreadsTable
  A routing table that maps account or message routes to thread identifiers.

• ThreadIdentifier
  An identifier for a processing thread within the system.

• AccountRouting
  Composed of a DApp identifier and an account address used to determine routing.

• OutMsgQueue
  Represents a queue of outgoing messages from accounts in the shard.

• AccountId and AccountAddress
  Identifiers used for accounts within the shard.

• get_account_routing_for_account
  Function (imported from multithreading::account) that computes the routing information for a given account.

Interaction with Other Parts of the System

• Multithreading Module:
  The cropping logic relies on the get_account_routing_for_account function from the multithreading::account module to determine routing for accounts.

• Types Module:
  The file uses several types (AccountAddress, AccountRouting, BlockIdentifier, DAppIdentifier, ThreadIdentifier, ThreadsTable) from the types module to manage routing and threading metadata.

• TVM Block and Types:
  The file uses types and traits (HashmapAugType, OutMsgQueue, ShardStateUnsplit, HashmapType, HashmapRemover) from tvm_block and tvm_types for managing account state and message queues.

• Tracing and Error Handling:
  The file uses the tracing crate for logging and anyhow for error wrapping and propagation.

Visual Diagram

flowchart TD
A[Input: ShardStateUnsplit] --> B[Read Accounts]
A --> C[Read Outgoing Messages Queue]
C --> D{Iterate Messages}
D -->|For each message| E[Check Destination Routing]
E --> F{Match Thread?}
F -->|No| G[Mark Message for Removal]
F -->|Yes| H[Keep Message]
G --> I[Remove Marked Messages]
B --> J{Optimization Skip Accounts Crop?}
J -->|No| K[Iterate Accounts]
K --> L[Get Account Routing]
L --> M{Match Thread?}
M -->|No| N[Mark Account for Removal or Redirect]
M -->|Yes| O[Keep Account]
N --> P[Remove or Replace Account]
J -->|Yes| O
I --> Q[Write Out Msg Queue]
P --> R[Write Accounts]
Q --> S[Return Filtered ShardState]
R --> S

This diagram shows the main workflow of the cropping operation, detailing message and account filtering paths and the final write-back to the shard state.

End of Documentation for shard_state_operations.rs