Probe Scripts

Purpose

Probe scripts serve as lightweight health checks for blockchain node daemons and related services running within Kubernetes pods. Their primary role is to provide liveness, readiness, and startup signals that help Kubernetes determine whether a container is functioning correctly, ready to serve requests, or requires restarting. This subtopic addresses the challenge of reliably detecting node synchronization status, network connectivity, and operational health in a heterogeneous blockchain environment where each chain has unique node RPC interfaces and synchronization semantics.

Unlike generic container health checks, these scripts implement blockchain-specific logic by querying node RPC endpoints to assess synchronization progress, peer connectivity, and block advancement. They ensure that only healthy and fully synced nodes receive traffic, maintaining overall system stability and data consistency throughout the multi-blockchain ShapeShift Unchained infrastructure.

Functionality

Probe scripts typically perform the following key workflows:

• Readiness Probe: Verifies if the node has caught up to the network’s latest block height within a configurable tolerance and has active peer connections. For example, the Bitcoin readiness script queries the getblockchaininfo and getconnectioncount RPC methods to compare local vs. network block heights and peer count. If synced and peers exist, it signals readiness.

• Liveness Probe: Checks if the node daemon is actively progressing by confirming that the current block number increases over time. The Ethereum liveness script stores the last observed block number in a temporary file and compares it to the current block number fetched via eth_blockNumber. If the block number advances, the daemon is alive; otherwise, it is considered stalled.

• Startup Probe (not detailed here): May perform initial synchronization checks or wait for a minimum number of blocks or peers before signaling startup completion.

Each script respects a disabling flag file (e.g., `/data/disable_readiness`) allowing manual override to bypass probe failures during maintenance or debugging.

Example: Bitcoin Readiness Check Snippet

CONNECTION_COUNT=$(curl -sf -H 'content-type: application/json' -u user:password \
  -d '{ "jsonrpc": "2.0", "id": "probe", "method": "getconnectioncount", "params": [] }' http://localhost:8332) || exit 1
BLOCKCHAIN_INFO=$(curl -sf -H 'content-type: application/json' -u user:password \
  -d '{ "jsonrpc": "2.0", "id": "probe", "method": "getblockchaininfo", "params": [] }' http://localhost:8332) || exit 1

PEERS=$(echo $CONNECTION_COUNT | jq -r '.result')
NODE_LATEST_BLOCK_HEIGHT=$(echo $BLOCKCHAIN_INFO | jq -r '.result.blocks')
NETWORK_LATEST_BLOCK_HEIGHT=$(echo $BLOCKCHAIN_INFO | jq -r '.result.headers')

if (( NODE_LATEST_BLOCK_HEIGHT >= NETWORK_LATEST_BLOCK_HEIGHT - 1 )) && (( PEERS > 0 )); then
  exit 0
else
  exit 1
fi

Key Characteristics

• Use of blockchain RPC APIs for node status data.

• JSON parsing with jq to extract relevant fields.

• Exit codes 0 (success) / 1 (failure) to signal probe result to Kubernetes.

• Conditional logic tailored per blockchain based on available RPC methods and synchronization nuances.

• Minimal external dependencies to ensure fast execution and low resource usage.

Integration

Probe scripts integrate tightly with Kubernetes deployment manifests for blockchain daemon StatefulSets and pods. Kubernetes specifies these scripts as the command for readiness, liveness, and startup probes in pod spec containers, enabling automated health monitoring and lifecycle management.

These scripts complement other subtopics such as:

• Infrastructure Scripts: Probe scripts form part of the broader shell script utilities managing daemon lifecycle and health.

• Deployment Automation: Pulumi deployment scripts reference probe scripts to configure Kubernetes readiness and liveness probes dynamically during coinstack deployment.

• Multi-Blockchain Coinstacks: Each blockchain implementation provides its own variant of probe scripts adapted to its node RPC interface and health semantics.

• Monitoring: Probe failure events trigger Kubernetes pod restarts, which are surfaced via Prometheus metrics and Grafana dashboards, facilitating proactive alerting.

By providing chain-specific health checks, probe scripts enable the unified API layer and indexers to rely on healthy node data sources, improving overall system robustness and user experience.

Probe Script Health Check Flow

flowchart TD
    Start[Start Probe Script]
    CheckDisable{Disable Flag Present?}
    FetchRPC[Fetch Node RPC Data]
    ParseData[Parse and Extract Status]
    EvaluateStatus{Is Node Synced and Healthy?}
    Success[Exit 0 - Healthy]
    Failure[Exit 1 - Unhealthy]

    Start --> CheckDisable
    CheckDisable -- Yes --> Success
    CheckDisable -- No --> FetchRPC
    FetchRPC --> ParseData
    ParseData --> EvaluateStatus
    EvaluateStatus -- Yes --> Success
    EvaluateStatus -- No --> Failure

This flowchart depicts the typical probe script logic: it first checks for a disable flag to optionally bypass health checks, then queries the node’s RPC interface, parses the response, evaluates synchronization and peer status, and finally exits with a code indicating health to Kubernetes.