liveness.sh

Overview

`liveness.sh` is a Bash script designed to serve as a **liveness probe** for a daemon process that interacts with an Ethereum node via JSON-RPC. Its primary function is to detect whether the daemon is actively processing blockchain data by checking if the Ethereum block number is progressing over time.

The script periodically queries the Ethereum node running locally, compares the current block number with the previously recorded block number, and determines if the daemon is still running (i.e., not stalled). It exits with a status code indicating the health status, which can be used by container orchestration systems (e.g., Kubernetes) to restart or manage the daemon accordingly.

Additionally, the script supports a mechanism to disable the liveness check temporarily by the presence of a specific "disable flag" file.

Detailed Explanation

Constants and Files

Script Logic Flow

1. Check for Disable Flag

   if [[ -f "$DISABLE_LIVENESS_PROBE" ]]; then
     echo "liveness probe disabled"
     exit 0
   fi
   

   • If the file /data/disable_liveness exists, the script immediately exits with 0 (success), effectively disabling the liveness check.

2. Query Ethereum Node for Current Block Number

   ETH_BLOCK_NUMBER=$(curl -sf -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' -H 'Content-Type: application/json' http://localhost:8545) || exit 1
   

   • Sends a JSON-RPC POST request to the Ethereum node on localhost:8545, calling the eth_blockNumber method.

   • On failure (curl error or no response), the script exits with status 1 indicating failure.

3. Parse and Convert Block Number

   CURRENT_BLOCK_NUMBER_HEX=$(echo $ETH_BLOCK_NUMBER | jq -r '.result')
   CURRENT_BLOCK_NUMBER=$(($CURRENT_BLOCK_NUMBER_HEX))
   

   • The Ethereum node returns the block number as a hexadecimal string (e.g., "0x10d4f").

   • The script uses jq to extract the .result field.

   • Converts the hex string to a decimal integer using shell arithmetic.

4. Initialize Block Number File if Missing

   if [[ ! -f "$FILE" ]]; then
     echo $CURRENT_BLOCK_NUMBER > $FILE
     exit 1
   fi
   

   • If the file storing the previous block number does not exist, it is created with the current block number.

   • The script exits with status 1, indicating the daemon's liveness cannot yet be confirmed (first run).

5. Compare Current and Previous Block Numbers

   PREVIOUS_BLOCK_NUMBER=$(cat $FILE)
   echo $CURRENT_BLOCK_NUMBER > $FILE
   
   if (( $CURRENT_BLOCK_NUMBER > $PREVIOUS_BLOCK_NUMBER )); then
     echo "daemon is running"
     exit 0
   fi
   
   echo "daemon is stalled"
   exit 1
   

   • Reads the previously stored block number.

   • Updates the file with the current block number.

   • If the current block number is greater than the previous, it signals progress and exits with success (0).

   • Otherwise, it concludes the daemon is stalled and exits with failure (1).

Usage Example

Run the script directly from the command line or configure it as a Kubernetes liveness probe:

./liveness.sh

• If the script outputs "daemon is running" and exits with 0, the probe is successful.

• If the script outputs "daemon is stalled" or encounters errors, it exits with 1, indicating the daemon may require a restart.

Implementation Details and Algorithms

• Liveness Detection: The core algorithm relies on detecting increments in the Ethereum block number, which implies the daemon is processing blockchain data and not stuck.

• Use of JSON-RPC: The script uses Ethereum's JSON-RPC interface to query the node locally.

• Hexadecimal Handling: Since the Ethereum JSON-RPC returns block numbers in hex, the script converts this to decimal for easy numeric comparison.

• State Persistence: Uses a file /data/.block_number to persist the last seen block number across script runs.

• Disable Mechanism: Allows temporarily skipping the liveness check by presence of a flag file, useful during maintenance or upgrades.

Interaction with Other System Components

• Ethereum Node: The script depends on an Ethereum node running on localhost:8545. It queries this node to retrieve the current block number.

• File System: Reads and writes files in /data, which must be writable and persistent to maintain state between probes.

• Orchestration Platform (e.g., Kubernetes): Intended as a liveness probe script that can be periodically executed by a Kubernetes probe or similar health check mechanism.

• Daemon Process: The daemon whose liveness is monitored is assumed to be the one advancing the blockchain state or interacting with the Ethereum node.

Visual Diagram

flowchart TD
    A[Start liveness.sh] --> B{Disable flag file?}
    B -- Yes --> C[Print "liveness probe disabled" and exit 0]
    B -- No --> D[Query Ethereum node via JSON-RPC]
    D --> E{Query successful?}
    E -- No --> F[Exit 1]
    E -- Yes --> G[Parse block number from hex to decimal]
    G --> H{Block number file exists?}
    H -- No --> I[Write current block number to file and exit 1]
    H -- Yes --> J[Read previous block number from file]
    J --> K[Write current block number to file]
    K --> L{Current > Previous?}
    L -- Yes --> M[Print "daemon is running" and exit 0]
    L -- No --> N[Print "daemon is stalled" and exit 1]

Summary

• liveness.sh is a lightweight, robust liveness probe script for Ethereum daemon health monitoring.

• It verifies daemon activity by checking that the Ethereum block number is increasing.

• Supports disabling via a flag file.

• Suitable for integration in container orchestration environments to automate daemon restarts upon failure detection.

• Utilizes standard tools (curl, jq) and simple file-based state management.

This approach ensures minimal dependencies and straightforward integration with existing Ethereum-based systems.