cross_thread_ref_repository.rs

Overview

This file implements a repository for managing CrossThreadRefData, which represents cross-thread reference data associated with blockchain blocks identified by BlockIdentifier. It is designed to operate efficiently in multithreaded environments, supporting lookup, caching, and persistence of cross-thread reference data. The repository provides methods to retrieve individual cross-thread reference data items as well as a historical tail of such data for up to 100 blocks, facilitating state synchronization during node join operations.

The file relies on an LRU cache (LruCache) to optimize access to frequently requested data and supports two modes of data storage: file-based and database-backed (controlled by the messages_db feature flag). It uses synchronization primitives (Mutex, Arc) from the parking_lot crate to ensure thread-safe access to the cache.

Traits

CrossThreadRefDataRead

pub trait CrossThreadRefDataRead {
    fn get_cross_thread_ref_data(
        &self,
        identifier: &BlockIdentifier,
    ) -> anyhow::Result<CrossThreadRefData>;
}

• Purpose: Abstracts the ability to read CrossThreadRefData for a given block identifier.

• Method:

  • get_cross_thread_ref_data: Retrieves the cross-thread reference data for the specified block.

• Parameters:

  • identifier: A reference to a BlockIdentifier which uniquely identifies a block.

• Returns: anyhow::Result<CrossThreadRefData> containing the data on success, or an error if not found or on failure.

CrossThreadRefDataHistory

pub trait CrossThreadRefDataHistory {
    fn get_history_tail(
        &self,
        identifier: &BlockIdentifier,
    ) -> anyhow::Result<Vec<CrossThreadRefData>>;
}

• Purpose: Provides access to a tail of historical cross-thread reference data, useful for reconstructing recent block histories.

• Method:

  • get_history_tail: Retrieves up to 100 blocks of cross-thread reference data history, starting from the given block identifier and traversing parent blocks backward.

• Parameters:

  • identifier: The starting BlockIdentifier for the history tail.

• Returns: A vector of CrossThreadRefData ordered from the starting block backward, or an error on failure.

Structs

CrossThreadRefDataRepository

#[derive(Clone)]
pub struct CrossThreadRefDataRepository {
    data_dir: PathBuf,
    cross_thread_ref_data_cache: Arc<Mutex<LruCache<BlockIdentifier, Option<CrossThreadRefData>>>>,
    crossref_store: CrossRefStorage,
}

• Purpose: Concrete implementation of the repository managing cross-thread reference data with caching and persistent storage.

• Fields:

  • data_dir: Directory path where cross-thread reference data files are stored.

  • cross_thread_ref_data_cache: Thread-safe, shared LRU cache mapping BlockIdentifier to optional CrossThreadRefData.

  • crossref_store: Abstraction over storage backend (file or database).

Implementation of CrossThreadRefDataRead

• Method: get_cross_thread_ref_data

  Retrieves the cross-thread reference data for a block, first checking the cache, then loading from persistent storage if necessary.

  • Locks the cache mutex to check for cached data.

  • If cached data exists and is Some, returns a clone.

  • If cached data is None, returns an error indicating data was not set.

  • If not cached, constructs the data file path using get_cross_thread_ref_data_path.

  • Depending on the build feature messages_db, reads the data from either crossref_store or file system.

  • Updates the cache with the loaded data or None if missing.

  • Returns the loaded data or an error if none found.

Implementation of CrossThreadRefDataHistory

• Method: get_history_tail

  Collects a history tail of cross-thread reference data starting from the given block identifier, traversing parents up to 100 blocks or until default block identifier is reached.

  • Starts by retrieving the data for the initial block identifier.

  • Iteratively attempts to fetch the parent block's cross-thread reference data.

  • Logs a trace message if data is missing and breaks the loop.

  • Terminates early if the parent is the default (null) block identifier.

  • Returns a vector of collected CrossThreadRefData instances.

Other Methods

• Constructor: new

  pub fn new(
      data_dir: PathBuf,
      thread_cnt_soft_limit: usize,
      crossref_store: CrossRefStorage,
  ) -> Self
  

  • Initializes a new CrossThreadRefDataRepository.

  • Sets up the LRU cache size as CROSS_THREAD_REF_DATA_CACHE_SIZE * thread_cnt_soft_limit.

  • Stores the data directory and cross reference storage instance.

• Helper: get_cross_thread_ref_data_path

  fn get_cross_thread_ref_data_path(&self, block_id: &BlockIdentifier) -> PathBuf
  

  • Constructs the file path for the cross-thread reference data of the given block.

  • Uses hexadecimal string representation of the block identifier.

  • Joins with "cross-thread-ref-data" directory under the base data_dir.

• Mutator: set_cross_thread_ref_data

  pub fn set_cross_thread_ref_data(
      &mut self,
      cross_thread_ref_data: CrossThreadRefData,
  ) -> anyhow::Result<()>
  

  • Stores cross-thread reference data persistently and updates the cache.

  • Checks if data for the block identifier is already cached and present, returns early if so.

  • Determines the file path for saving.

  • Computes total outbound messages count for tracing.

  • Depending on the messages_db feature, writes data either to the crossref_store or filesystem.

  • Updates the cache with the newly stored data.

  • Uses tracing instrumentation to log detailed metrics during save operation.

Important Implementation Details

• Cache Size and Thread Count

  The cache size is dynamically scaled by the thread_cnt_soft_limit parameter multiplied by a constant CROSS_THREAD_REF_DATA_CACHE_SIZE (10,000), allowing efficient caching in multithreaded environments.

• Thread Safety

  The cache is wrapped inside an Arc<Mutex<>> to allow safe concurrent access from multiple threads without expensive blocking primitives.

• Configurable Storage Backend

  The file supports two backends for persistent storage controlled by the messages_db feature flag:

  • File system: Uses load_from_file and save_to_file functions.

  • Database: Uses CrossRefStorage methods read_blob and write_blob.

• History Retrieval Algorithm

  The get_history_tail method retrieves a chain of cross-thread reference data by following parent block identifiers recursively, with a hardcoded limit of 100 blocks to avoid unbounded traversal.

• Error Handling

  Uses anyhow::Result for error handling, with early returns on cache misses or file/db errors. Panics only occur if the file/db loading unexpectedly fails, indicating critical data corruption or filesystem issues.

• Instrumentation

  The set_cross_thread_ref_data method is instrumented with tracing macros to provide detailed runtime telemetry, including counts of outbound messages and block metadata.

Interaction with Other System Components

• CrossThreadRefData

  This file depends on the CrossThreadRefData data structure which encapsulates the data referenced by blocks.

• BlockIdentifier

  Uses BlockIdentifier as a unique key to identify blocks and their associated cross-thread references.

• CrossRefStorage

  Abstracts the underlying storage mechanism for cross-thread reference data. This repository uses it for reading and writing blobs when the messages_db feature is enabled.

• repository_impl

  Imports utility functions load_from_file and save_to_file for file-based persistence.

• Thread Join Operation

  The repository supports operations needed during a node join in a multi-threaded environment by providing historical cross-thread reference data tails.

Visual Diagram: Class Structure and Method Relationships

classDiagram
class CrossThreadRefDataRepository {
-data_dir: PathBuf
-cross_thread_ref_data_cache: Arc<Mutex<LruCache<BlockIdentifier, Option<CrossThreadRefData>>>>
-crossref_store: CrossRefStorage
+new()
+get_cross_thread_ref_data_path()
+set_cross_thread_ref_data()
+get_cross_thread_ref_data()
+get_history_tail()
}
CrossThreadRefDataRepository ..|> CrossThreadRefDataRead
CrossThreadRefDataRepository ..|> CrossThreadRefDataHistory

This diagram shows the main struct CrossThreadRefDataRepository with its fields and key methods, illustrating its implementation of the two traits for reading and retrieving history. The private helper method for path construction and the mutator method for setting data are included to show internal workflow.