liveness.sh

Overview

`liveness.sh` is a lightweight Bash script designed to act as a Kubernetes-style liveness probe for a daemon interacting with an Ethereum node. The script checks if the daemon is actively processing new Ethereum blocks by comparing the current block number retrieved via an Ethereum JSON-RPC call against a previously recorded block number. If the block number has advanced, the script considers the daemon healthy (alive); if not, it reports the daemon as stalled.

The script also supports a toggle to disable the liveness check via a specific file, allowing manual override in certain operational scenarios.

Detailed Explanation

Purpose

To monitor the health of a daemon that depends on Ethereum blockchain progress.
To detect stalled states that may require container restart or alerting.
To provide an opt-out mechanism for disabling the liveness probe temporarily.

Script Workflow and Behavior

Step	Action	Description
1	Check if liveness probe is disabled	If `/data/disable_liveness` exists, print a message and exit `0` (success), skipping further checks.
2	Retrieve current Ethereum block number	Perform a JSON-RPC call to the local Ethereum node at `http://localhost:8545` to get the current block number in hexadecimal format. Exit with failure if this call fails.
3	Convert block number from hex to decimal	Extract the `.result` field from the JSON response using `jq` and convert it to a decimal integer.
4	Check for existence of stored block number file	If `/data/.block_number` does not exist, write the current block number to the file and exit with failure (`1`) to indicate the probe is not yet ready.
5	Compare current block number with previous	Read the previous block number from the file, update the file with the current block number, and check if the current number is greater than the previous.
6	Determine health	If the block number has increased, print "daemon is running" and exit `0`; otherwise, print "daemon is stalled" and exit `1`.

Important Variables

Variable	Description
DISABLE_LIVENESS_PROBE	Path to the file that disables the liveness probe if it exists (`/data/disable_liveness`).
FILE	Path to the file storing the last known Ethereum block number (`/data/.block_number`).
ETH_BLOCK_NUMBER	Raw JSON response from the Ethereum node's `eth_blockNumber` JSON-RPC call.
CURRENT_BLOCK_NUMBER_HEX	Extracted block number in hexadecimal string form.
`CURRENT_BLOCK_NUMBER`	Current block number converted to decimal integer.
`PREVIOUS_BLOCK_NUMBER`	Previously stored block number read from file.

Usage Examples

This script is typically used as a liveness probe command in a Kubernetes Pod specification for a container running the daemon:

livenessProbe:
  exec:
    command:
      - /bin/bash
      - /path/to/liveness.sh
  initialDelaySeconds: 15
  periodSeconds: 10

**Manual invocation:**

bash liveness.sh

Expected output and exit codes:

If the probe is disabled:
Output: liveness probe disabled
Exit code: 0 (success)
If the block number has progressed:
Output: daemon is running
Exit code: 0 (success)
If the block number has stalled:
Output: daemon is stalled
Exit code: 1 (failure)
On first run when the block number file does not exist:
No previous block to compare, file created with current block number.
Exit code: 1 (failure) to indicate transient state.

Implementation Details & Algorithms

Ethereum JSON-RPC Call:
Uses curl with POST data to call eth_blockNumber method of the local Ethereum node. The call is silent on failure (-sf), and the script exits immediately on failure to ensure probe accuracy.
Block Number Parsing:
Extracts the block number from the JSON .result field using jq. The block number is in hexadecimal (e.g., "0x10d4f"), which is converted into a decimal integer using shell arithmetic.
State Persistence:
Stores the last observed block number in a hidden file (/data/.block_number). This file is used to compare block progression across probe runs.
Health Check Logic:
The core liveness check is a simple numeric comparison:
CURRENT_BLOCK_NUMBER > PREVIOUS_BLOCK_NUMBER means the daemon has processed new blocks and is healthy.
Otherwise, it may be stalled.
Disabling Probe:
By checking for the presence of a file (/data/disable_liveness), the script supports temporary disabling of the liveness check, useful during maintenance or troubleshooting.

Interaction with Other System Components

Ethereum Node (Local RPC at localhost:8545):
The script depends on communication with the Ethereum node's JSON-RPC API to obtain blockchain progress.
Daemon Being Monitored:
The assumption is that this daemon processes Ethereum blocks in real-time or near real-time. The liveness is inferred from the daemon's ability to track new blocks.
Kubernetes / Container Orchestration:
Typically integrated into container liveness probes for automated health checking and restarting of failed containers.
Persistent Storage (/data directory):
The script reads and writes to files in /data volume mount, which should be persistent across probe executions.

Mermaid Flowchart Diagram: Script Workflow

flowchart TD
    A[Start] --> B{Disable Liveness File Exists?}
    B -- Yes --> C[Print "liveness probe disabled"]
    C --> D[Exit 0]
    B -- No --> E[Call Ethereum JSON-RPC eth_blockNumber]
    E --> F{Call Successful?}
    F -- No --> G[Exit 1]
    F -- Yes --> H[Parse block number (hex -> dec)]
    H --> I{Block Number File Exists?}
    I -- No --> J[Write current block number to file]
    J --> K[Exit 1]
    I -- Yes --> L[Read previous block number]
    L --> M[Write current block number to file]
    M --> N{Current > Previous?}
    N -- Yes --> O[Print "daemon is running"]
    O --> P[Exit 0]
    N -- No --> Q[Print "daemon is stalled"]
    Q --> R[Exit 1]

Summary

`liveness.sh` is a simple yet effective health check script that leverages Ethereum blockchain progress to monitor the state of a blockchain-processing daemon. Its use of persistent state and JSON-RPC querying ensures accurate detection of stalled conditions, making it suitable for containerized environments requiring automatic recovery of unresponsive services. The script’s structure is straightforward, enabling easy customization or extension as needed.

**End of documentation for `liveness.sh`.**