liveness.sh

Overview

`liveness.sh` is a Bash script designed to act as a **liveness probe** for a daemon process that interacts with an Ethereum-compatible blockchain node (specifically Avalanche C-Chain RPC). The script checks whether the daemon is actively progressing by monitoring the increase in the blockchain's block number over time.

The script achieves this by querying the current block number from the local RPC endpoint and comparing it to a previously recorded block number stored on disk. If the block number has increased since the last check, the daemon is considered healthy (running). If the block number stagnates or the probe is disabled via a file flag, appropriate exit statuses and messages are returned to indicate the daemon’s health.

This liveness probe can be integrated with container orchestration platforms (e.g., Kubernetes) to automatically restart or alert if the daemon becomes unresponsive.

File Functionality and Workflow

1. Disable Condition:
   If the file /data/disable_liveness exists, the script immediately exits with status 0 indicating the probe is disabled and reports "liveness probe disabled".

2. Fetch Current Block Number:
   The script sends a JSON-RPC request to http://localhost:9650/ext/bc/C/rpc to invoke the eth_blockNumber method, which returns the current highest block number on the blockchain as a hex string.

3. First Run Initialization:
   If the file /data/.block_number does not exist, the script writes the current block number to this file and exits with status 1, signaling that this is the initial setup and no health determination can be made yet.

4. Block Number Comparison:
   The script reads the previously stored block number, compares it to the current block number, and updates the stored value.

5. Health Determination:

   • If the current block number is greater than the previous number, the daemon is deemed healthy, and the script exits with status 0.

   • If not, the daemon is considered stalled, and the script exits with status 1.

Detailed Explanation of Script Sections

Variables

• DISABLE_LIVENESS_PROBE=/data/disable_liveness
  Path to the file that disables the liveness probe when present.

• FILE=/data/.block_number
  File used to persist the last observed block number.

• ETH_BLOCK_NUMBER
  Holds the raw JSON-RPC response from the blockchain node.

• CURRENT_BLOCK_NUMBER_HEX
  Extracts the hex string block number from the JSON response.

• CURRENT_BLOCK_NUMBER
  Converts the hex string to a decimal number for comparison.

• PREVIOUS_BLOCK_NUMBER
  The block number read from the persistent file for comparison.

Key Commands and Logic

• curl -sf -d '{...}' -H 'Content-Type: application/json' http://localhost:9650/ext/bc/C/rpc
  Performs a silent (-s) and fail-on-error (-f) JSON-RPC POST request to query the current block number.

• jq -r '.result'
  Parses the JSON response to extract the result field (the hex block number).

• $(($CURRENT_BLOCK_NUMBER_HEX))
  Converts the hexadecimal block number string to a decimal integer.

• File existence checks ([[ -f "$FILE" ]]) determine initialization or disable conditions.

• Numeric comparison ((( $CURRENT_BLOCK_NUMBER > $PREVIOUS_BLOCK_NUMBER ))) to detect progress.

Usage Example

This script is typically executed as part of a container or process liveness check:

./liveness.sh

• If the block height is advancing, the script outputs:

daemon is running

and exits with status `0`.

• If the block height is not advancing:

daemon is stalled

and exits with status `1`.

• If the probe is disabled by presence of /data/disable_liveness:

liveness probe disabled

and exits with status `0`.

• On the first run (no previous block number stored), it stores the current block number and exits with status 1 to indicate no conclusive health status yet.

Important Implementation Details

• Persistence Mechanism:
  The script uses a simple file-based persistence (/data/.block_number) to store the last observed block number. This allows the script to detect if the daemon is progressing between invocations.

• Hexadecimal to Decimal Conversion:
  Ethereum JSON-RPC returns block numbers as hex strings (e.g., "0x10d4f"). The script converts this hex string to a decimal integer to perform numeric comparisons.

• Fail-Fast on RPC Failure:
  The curl command uses the -f flag, so if the RPC call fails (e.g., node is down), the script exits immediately with status 1, signaling an unhealthy state.

• Disable Flag:
  The probe can be disabled dynamically by creating the file /data/disable_liveness. This is useful for maintenance windows or debugging.

Interaction with Other System Components

• The script relies on the Ethereum-compatible node running locally on port 9650, specifically the Avalanche C-Chain RPC endpoint.

• It interacts with the filesystem under /data for storing state and control flags.

• Intended to be used by container orchestrators or monitoring tools as a health check command, influencing restart or alerting behavior.

• Could be integrated with Kubernetes liveness probes by specifying this script as the check command.

Mermaid Flowchart Diagram

flowchart TD
    A[Start liveness.sh] --> B{Is /data/disable_liveness present?}
    B -- Yes --> C[Print "liveness probe disabled" and exit 0]
    B -- No --> D[Send JSON-RPC request to get current block number]
    D --> E{Request successful?}
    E -- No --> F[Exit 1 (unhealthy)]
    E -- Yes --> G[Parse hex block number to decimal]
    G --> H{Does /data/.block_number exist?}
    H -- No --> I[Store current block number and exit 1]
    H -- Yes --> J[Read previous block number from file]
    J --> K[Store current block number to file]
    K --> L{Is current block number > previous?}
    L -- Yes --> M[Print "daemon is running" and exit 0]
    L -- No --> N[Print "daemon is stalled" and exit 1]

Summary

`liveness.sh` is a lightweight, file-backed liveness probe script tailored for a blockchain daemon that monitors the progress of block height via a local JSON-RPC call. It provides a mechanism to detect stalled states and disable probing when necessary, making it suitable for integration in containerized environments to improve system reliability and observability.