liveness.sh

Overview

`liveness.sh` is a Bash script designed as a Kubernetes liveness probe for the Heimdall service. Its primary purpose is to monitor whether the Heimdall node is actively processing new blocks in a blockchain environment. The script achieves this by periodically querying the node’s status endpoint and checking if the latest block height has advanced since the last check.

If the block height increases, the script concludes that Heimdall is running normally. If the height remains stagnant or if the status cannot be retrieved, the script signals an unhealthy state, indicating the node might be stalled. Additionally, the script supports a manual override to disable the liveness probe via a specific file.

Detailed Explanation

Variables and Constants

• DISABLE_LIVENESS_PROBE=/root/disable_liveness
  Path to a file which, if present, disables the liveness probe check.

• FILE=/root/.latest_block_height
  Path to the file that stores the last observed block height between runs.

Script Workflow

1. Check for Disable File
   If the file /root/disable_liveness exists, the script immediately prints "liveness probe disabled" and exits with status 0 (success). This allows manual disabling of the probe.

2. Retrieve Latest Block Height
   The script calls the Heimdall RPC status endpoint at http://localhost:26657/status using curl.

   • If the request fails (non-2xx response or connection issue), the script exits with status 1 indicating failure.

   • If successful, it parses the JSON response using jq to extract .result.sync_info.latest_block_height.

3. Initialize or Compare Block Height

   • If the file /root/.latest_block_height does not exist, the script writes the current latest block height to this file and exits with status 1, signaling the initial run or lack of baseline data.

   • If the file exists, it reads the previous block height, updates the file with the current height, and compares the two.

4. Determine Node Health

   • If the current block height is greater than the previous, the script prints "heimdall is running" and exits with status 0 (healthy).

   • Otherwise, it prints "heimdall is stalled" and exits with status 1 (unhealthy).

Important Implementation Details

• The script relies on curl for HTTP requests and jq for JSON parsing. Both must be installed and available in the container or environment where this script runs.

• Block height comparison uses integer arithmetic ((( ... ))) ensuring numeric comparison rather than string comparison.

• Exit codes:

  • 0 indicates the liveness probe passed (service healthy or disabled).

  • 1 indicates failure (service stalled or unable to retrieve status).

• The script stores state persistently by writing the last observed block height to a file in /root/.latest_block_height. This allows comparison across probe invocations.

Usage Example

This script is typically used as a Kubernetes liveness probe command in a Pod specification, for example:

livenessProbe:
  exec:
    command:
      - /bin/bash
      - /path/to/liveness.sh
  initialDelaySeconds: 30
  periodSeconds: 15

In this context, Kubernetes executes the script periodically. If the script exits with `0`, Kubernetes considers the container healthy; if it exits with non-zero, Kubernetes may restart the container.

Interaction with Other System Components

• Heimdall Node: The script queries Heimdall’s local RPC endpoint (localhost:26657) to retrieve the current blockchain sync status.

• Kubernetes: Acts as a liveness probe to Kubernetes, enabling automatic restart of the Heimdall Pod if it becomes unresponsive or stalled.

• Filesystem: Uses local disk state to persist last known block height and optionally to disable the probe.

• Utilities: Depends on curl and jq utilities installed in the container.

Flowchart: Script Workflow

flowchart TD
    Start([Start])
    CheckDisable{Is disable file\n(/root/disable_liveness)\n present?}
    PrintDisabled["Print 'liveness probe disabled'\nExit 0"]
    CurlStatus["Call http://localhost:26657/status\nusing curl"]
    CurlFail{Did curl succeed?}
    ReadLatestBlock["Parse latest_block_height\nfrom JSON response"]
    CheckFileExist{Does /root/.latest_block_height\nfile exist?}
    WriteInitial["Write latest_block_height\nto file\nExit 1"]
    ReadPreviousBlock["Read previous block height\nfrom file"]
    UpdateFile["Update file with\nlatest_block_height"]
    CompareBlocks{Is latest_block_height >\nprevious_block_height?}
    Healthy["Print 'heimdall is running'\nExit 0"]
    Stalled["Print 'heimdall is stalled'\nExit 1"]

    Start --> CheckDisable
    CheckDisable -- Yes --> PrintDisabled
    CheckDisable -- No --> CurlStatus
    CurlStatus --> CurlFail
    CurlFail -- No --> Stalled
    CurlFail -- Yes --> ReadLatestBlock
    ReadLatestBlock --> CheckFileExist
    CheckFileExist -- No --> WriteInitial
    CheckFileExist -- Yes --> ReadPreviousBlock
    ReadPreviousBlock --> UpdateFile
    UpdateFile --> CompareBlocks
    CompareBlocks -- Yes --> Healthy
    CompareBlocks -- No --> Stalled

Summary

This script provides a lightweight, effective mechanism for Kubernetes to monitor the health of the Heimdall node by tracking blockchain progress. It uses a simple file-based state and HTTP status queries, making it easy to deploy and maintain within containerized environments. The ability to disable the probe via a file adds operational flexibility for maintenance or troubleshooting.

End of Documentation for liveness.sh