Daemon Probes

Purpose

Daemon Probes are specialized shell scripts designed to verify the health and synchronization status of blockchain daemon nodes. These probes address the critical problem of ensuring that each blockchain node is fully synchronized with its network and maintains active peer connections before being declared ready to serve requests. This readiness check is essential within the broader health and readiness probes framework to maintain reliability and uptime of blockchain services.

Functionality

The core functionality of Daemon Probes revolves around three key checks:

1. Node Synchronization Status:
   Each probe queries the daemon node’s RPC interface to determine the node’s current block height and compares it against the latest known network block height, allowing a configurable tolerance. This ensures the node’s blockchain state is near or at the latest network state.

2. Peer Connectivity:
   The probes verify that the node is connected to a sufficient number of peers, indicating active participation in the P2P network and ongoing data propagation.

3. Conditional Probe Disabling:
   Probes support disabling via presence of a sentinel file (/data/disable_readiness), allowing manual override or maintenance modes without triggering readiness failures.

Blockchain-Specific Workflows

• Bitcoin and Litecoin Daemon Probes:
  Use JSON-RPC calls getconnectioncount and getblockchaininfo to retrieve peer count and block heights. The comparison subtracts a tolerance (e.g., 1 block) from the network’s latest block header to allow slight lag. If synchronized and peers exist, the node is marked ready.

• Ethereum Daemon Probe:
  Uses eth_syncing RPC method to check sync status. If not syncing, it fetches the current block number and compares it against an aggregated best reference block number obtained from multiple external RPC endpoints. A tolerance buffer (e.g., 5 blocks) is applied to determine readiness.

Example Snippet (Bitcoin / Litecoin readiness check):

CONNECTION_COUNT=$(curl -sf ... -d '{ "method": "getconnectioncount" }')
BLOCKCHAIN_INFO=$(curl -sf ... -d '{ "method": "getblockchaininfo" }')

PEERS=$(echo $CONNECTION_COUNT | jq -r '.result')
NODE_BLOCK=$(echo $BLOCKCHAIN_INFO | jq -r '.result.blocks')
NETWORK_BLOCK=$(echo $BLOCKCHAIN_INFO | jq -r '.result.headers')

if (( NODE_BLOCK >= NETWORK_BLOCK - TOLERANCE )) && (( PEERS > 0 )); then
  echo "node is synced with $PEERS peers"
  exit 0
fi

Integration with Parent Topic and Other Subtopics

• Relationship to Health and Readiness Probes:
  Daemon Probes form one half of the readiness and liveness verification system, focusing exclusively on the blockchain daemon nodes. They complement Indexer Probes, which perform similar checks on Blockbook indexers to ensure the entire data pipeline from blockchain node to API is healthy.

• Interaction with Kubernetes Probes:
  These scripts are invoked by Kubernetes readiness and liveness probes to automate pod lifecycle management based on actual blockchain node status, enabling Kubernetes to restart pods when nodes fall out of sync or disconnect.

• Support for Deployment Automation:
  Daemon Probes are integrated into the container images and Kubernetes StatefulSets deployed via Pulumi, ensuring that each daemon pod self-monitors its readiness before accepting traffic.

• Enhancement Over Common API Layer:
  While the common API abstracts blockchain data and interactions, Daemon Probes guarantee the underlying data source—the daemon node—is current and connected, thereby supporting consistent API responses.

This subtopic introduces the blockchain-specific logic for node readiness verification that is not covered by the general health checks or indexer readiness scripts. It ensures early detection of blockchain node issues to maintain system reliability.

Diagram

flowchart TD
  Start[Start Probe Script] --> CheckDisable{Disable File Present?}
  CheckDisable -- Yes --> ExitSuccess[Exit 0: Probe Disabled]
  CheckDisable -- No --> QueryRPC[Query Daemon RPC Endpoint]
  QueryRPC --> ParseData[Parse Response: Block Height, Peers, Sync Status]
  ParseData --> CompareBlocks{Is Node Block Height \n≥ Network Block Height - Tolerance?}
  CompareBlocks -- No --> ExitFail[Exit 1: Node Still Syncing]
  CompareBlocks -- Yes --> CheckPeers{Are Peers > 0?}
  CheckPeers -- Yes --> ExitSuccessReady[Exit 0: Node Synced & Connected]
  CheckPeers -- No --> ExitFailNoPeers[Exit 1: Synced but No Peers]

This flowchart illustrates the logical decision process a Daemon Probe script follows to determine if a blockchain node is ready. It shows the conditional checks for disabling the probe, data retrieval from the node, synchronization status validation, and peer connectivity verification before concluding readiness.