liveness.sh

Overview

`liveness.sh` is a lightweight Bash script designed as a Kubernetes liveness probe for a blockchain node daemon, specifically interacting with an Ethereum-compatible JSON-RPC endpoint. Its primary purpose is to verify if the blockchain daemon is actively progressing by monitoring the increment of the blockchain's latest block number. If the block number advances over time, the daemon is considered healthy ("running"); otherwise, it is flagged as stalled.

The script also supports a manual override to disable the liveness probe by placing a specific file (`/data/disable_liveness`), allowing administrators to temporarily bypass health checks.

Detailed Explanation

Purpose

• To provide Kubernetes or other orchestration systems with a simple health check mechanism.

• To detect if the Ethereum daemon is alive and producing new blocks.

• To allow disabling the liveness check when necessary.

Script Workflow

1. Disable Check
   Checks if the file /data/disable_liveness exists.

   • If yes, prints "liveness probe disabled" and exits with status 0 (success), skipping further checks.

2. Fetch Current Block Number
   Uses curl to send a JSON-RPC request to http://localhost:8545 to get the latest Ethereum block number (eth_blockNumber method).

   • If the request fails, the script exits with status 1 (failure).

3. Parse and Convert Block Number
   Extracts the block number in hexadecimal from the JSON response using jq, then converts it to a decimal integer.

4. First Run Initialization
   Checks if the file /data/.block_number exists.

   • If not, writes the current block number to this file and exits with status 1.

   • This ensures the script has a baseline block number for future comparisons.

5. Compare Block Numbers
   Reads the previous block number from /data/.block_number and updates it with the current block number.

   • If the current block number is greater than the previous, prints "daemon is running" and exits with status 0.

   • Otherwise, prints "daemon is stalled" and exits with status 1.

Important Variables and Files

Usage Example

This script is typically invoked as a Kubernetes liveness probe command:

livenessProbe:
  exec:
    command:
      - /bin/bash
      - /path/to/liveness.sh
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

**Manual disable example:** To disable the liveness probe temporarily (e.g., for maintenance):

touch /data/disable_liveness
# The next liveness check will print "liveness probe disabled" and succeed.

Remove the file to re-enable:

rm /data/disable_liveness

Implementation Details

• JSON-RPC call: The script uses a minimal curl POST request with JSON payload to get the block number from the local Ethereum node.

• Robustness:

  • Uses curl -sf to fail silently and return non-zero exit code if the node is unreachable or returns an error.

  • Uses jq to parse JSON safely.

• State persistence:

  • Stores the last known block number in a file to compare on subsequent runs.

• Exit codes:

  • 0 indicates healthy (running or disabled).

  • 1 indicates failure (stalled or initialization).

Interaction with Other System Components

• Ethereum Daemon: The script relies on the Ethereum node exposing its JSON-RPC interface locally on port 8545.

• Kubernetes / Container Runtime:

  • Used as a liveness probe script to inform the orchestrator about container health.

  • The exit status governs Kubernetes pod restarts or alerts.

• Persistent Storage: Uses files under /data/ directory, which should be backed by persistent storage volume to maintain state across container restarts.

Mermaid Flowchart Diagram

flowchart TD
    Start([Start])
    CheckDisable{Is /data/disable_liveness present?}
    ExitDisabled["Print 'liveness probe disabled'\nExit 0"]
    FetchBlock["Fetch block number via\ncurl JSON-RPC"]
    CurlFail{Curl success?}
    ParseBlock["Parse blockNumber from JSON\nConvert hex to decimal"]
    CheckFile{Does /data/.block_number exist?}
    InitFile["Write current block number\nto /data/.block_number\nExit 1"]
    ReadPrev["Read previous block number\nfrom /data/.block_number"]
    WriteCurr["Write current block number\nto /data/.block_number"]
    CompareBlocks{Is current block > previous?}
    Running["Print 'daemon is running'\nExit 0"]
    Stalled["Print 'daemon is stalled'\nExit 1"]

    Start --> CheckDisable
    CheckDisable -- Yes --> ExitDisabled
    CheckDisable -- No --> FetchBlock
    FetchBlock --> CurlFail
    CurlFail -- No --> Stalled
    CurlFail -- Yes --> ParseBlock
    ParseBlock --> CheckFile
    CheckFile -- No --> InitFile
    CheckFile -- Yes --> ReadPrev
    ReadPrev --> WriteCurr
    WriteCurr --> CompareBlocks
    CompareBlocks -- Yes --> Running
    CompareBlocks -- No --> Stalled

Summary

`liveness.sh` is a concise and effective script to monitor the health of an Ethereum node by checking block progression. It provides a simple mechanism for Kubernetes or other orchestrators to determine if the blockchain daemon is alive, with built-in support for disabling the probe and persisting state across runs. The script’s minimal dependencies and straightforward logic make it well-suited for containerized environments requiring reliable liveness checking.