liveness.sh

Overview

`liveness.sh` is a Bash script designed to act as a liveness probe for a daemon process interacting with a local Ethereum node (or compatible blockchain node). The script periodically verifies if the daemon is actively progressing by checking the latest block number reported by the node via JSON-RPC. If the block number increases between checks, the daemon is considered healthy (running). If the block number stalls, the script indicates a failure state.

Additionally, the script supports disabling the liveness probe via a special file flag, allowing manual override of the probe's behavior.

Detailed Explanation

Purpose

• To monitor the health of a blockchain daemon by checking the advancement of the latest block number.

• To support easy disabling of the liveness probe for maintenance or specific situations.

• To provide an exit code indicating the daemon status, suitable for integration with container orchestration platforms (e.g., Kubernetes liveness probes).

Script Flow and Logic

1. Disable Check

   The script first checks if the file /data/disable_liveness exists:

   • If it exists, the script prints "liveness probe disabled" and exits with code 0 (success), effectively skipping the liveness check.

2. Fetch Current Block Number

   • The script queries the local blockchain node running at http://localhost:8547 using a JSON-RPC call to method eth_blockNumber.

   • The response is parsed with jq to extract the block number in hexadecimal.

   • The hexadecimal number is converted to a decimal integer.

3. Block Number Comparison

   • The script stores the current block number in a file /data/.block_number.

   • If this file does not exist (first run or reset), it writes the current block number to this file and exits with code 1 indicating the daemon status cannot yet be confirmed.

   • On subsequent runs, the script reads the previous block number from the file.

   • If the current block number is greater than the previous one, the daemon is considered running; the script prints "daemon is running" and exits with 0.

   • If the current block number is not greater (equal or less), it indicates the daemon is stalled; the script prints "daemon is stalled" and exits with 1.

Important Variables

Exit Codes

Usage Example

Suppose you want to use `liveness.sh` as a Kubernetes liveness probe:

livenessProbe:
  exec:
    command:
      - /bin/bash
      - /path/to/liveness.sh
  initialDelaySeconds: 10
  periodSeconds: 15

This setup runs the script every 15 seconds and expects an exit code of 0 to indicate the daemon is alive.

Implementation Details

• Uses curl with silent and fail flags (-sf) to avoid noisy output if the node is unreachable.

• Uses jq to parse the JSON response safely.

• Stores the previous block number in a hidden file /data/.block_number, which must be writable by the script.

• The block number is returned in hexadecimal, so the script converts it to decimal using Bash arithmetic for comparison.

• The presence of /data/disable_liveness offers a manual override for temporarily disabling the probe.

Interaction with Other System Components

• Blockchain Node: The script communicates with the Ethereum-compatible node running on localhost:8547 via JSON-RPC.

• Filesystem: Reads and writes state to /data/.block_number and checks for /data/disable_liveness.

• Container Orchestration / Monitoring: Typically integrated as a liveness probe in container platforms like Kubernetes to automatically restart the daemon if it stalls.

Visual Diagram

flowchart TD
    A[Start] --> B{Is /data/disable_liveness present?}
    B -- Yes --> C[Print "liveness probe disabled"]
    C --> D[Exit 0]
    B -- No --> E[Send JSON-RPC request to localhost:8547]
    E --> F{Request success?}
    F -- No --> G[Exit 1]
    F -- Yes --> H[Parse block number from response]
    H --> I{Does /data/.block_number exist?}
    I -- No --> J[Write current block number to file]
    J --> K[Exit 1]
    I -- Yes --> L[Read previous block number]
    L --> M[Write current block number to file]
    M --> N{Is current block > previous block?}
    N -- Yes --> O[Print "daemon is running"]
    O --> P[Exit 0]
    N -- No --> Q[Print "daemon is stalled"]
    Q --> R[Exit 1]

Summary

The `liveness.sh` script is a critical utility for health monitoring of blockchain daemons by verifying block number progression. Its simple logic combined with JSON-RPC queries and state persistence enables reliable detection of daemon stalls, making it suitable for containerized environments requiring automated health checks.