readiness.sh

Overview

`readiness.sh` is a Bash script designed to serve as a readiness probe for a Heimdall node in a blockchain network environment. Its primary purpose is to determine if the Heimdall node is ready to serve requests based on its synchronization status and peer connectivity. This script is typically used in container orchestration platforms (like Kubernetes) to inform the orchestrator when the service is ready to handle traffic.

The script performs health checks by querying local RPC endpoints exposed by the Heimdall node, interpreting the responses to decide readiness:

It first checks if the readiness probe is explicitly disabled.
It then verifies synchronization status and peer connectivity.
Finally, it outputs appropriate messages and returns success or failure exit codes accordingly.

Detailed Explanation

Script Flow and Functionality

Disable Check
The script looks for the presence of a sentinel file /root/disable_readiness. If this file exists, the readiness probe is considered disabled, and the script exits successfully immediately.
```
DISABLE_READINESS_PROBE=/root/disable_readiness

if [[ -f "$DISABLE_READINESS_PROBE" ]]; then
  echo "readiness probe disabled"
  exit 0
fi
```
Fetch Node Status via RPC Calls
The script performs three HTTP GET requests to local RPC endpoints:
- http://localhost:1317/syncing — to check if the node is currently syncing.
- http://localhost:26657/net_info — to get network information including the number of connected peers.
- http://localhost:26657/status — to retrieve the node’s current status including catch-up state.
If any of these curl commands fail (non-zero exit code), the script exits with failure (exit code 1).
```
SYNCING=$(curl -sf http://localhost:1317/syncing) || exit 1
NET_INFO=$(curl -sf http://localhost:26657/net_info) || exit 1
STATUS=$(curl -sf http://localhost:26657/status) || exit 1
```
Parse JSON Responses
Using jq, the script extracts:
- syncing boolean from the /syncing endpoint.
- catching_up boolean from the /status endpoint.
- Number of peers n_peers from the /net_info endpoint.
```
IS_SYNCING=$(echo $SYNCING | jq -r '.syncing')
CATCHING_UP=$(echo $STATUS | jq -r '.result.sync_info.catching_up')
NUM_PEERS=$(echo $NET_INFO | jq -r '.result.n_peers')
```
Determine Node Readiness
The script considers the node ready if:
- It is not syncing (syncing == false)
- It is not catching up (catching_up == false)
- It has at least one peer connected (n_peers > 0)
If these conditions are met, the script outputs a success message and exits with code 0.
If the node is synced but has no peers, it is considered not ready (exits 1).
If the node is still syncing or catching up, it is considered not ready (exits 1).
```
if [[ $IS_SYNCING == false && $CATCHING_UP == false ]]; then
  if (( $NUM_PEERS > 0 )); then
    echo "heimdall is synced with $NUM_PEERS peers"
    exit 0
  fi

  echo "heimdall is synced, but has no peers"
  exit 1
fi

echo "heimdall is still syncing"
exit 1
```

Usage Example

This script is intended to be used as a Kubernetes readiness probe command:

readinessProbe:
  exec:
    command:
      - /bin/bash
      - /path/to/readiness.sh
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

When invoked, the script will:

Return exit code 0 if the Heimdall node is ready.
Return exit code 1 otherwise, indicating the pod should not receive traffic.

Important Implementation Details

Dependency on jq: The script uses jq to parse JSON responses. Ensure jq is installed in the container or environment where this script runs.
Localhost RPC Endpoints: The script assumes Heimdall exposes its REST API and RPC endpoints on localhost at ports 1317 and 26657.
Fail Fast on Curl Errors: Any failure to fetch the required endpoints results in immediate exit with code 1 to indicate unready status.
Peer Connectivity Check: The node must have at least one connected peer to be considered ready, which reflects actual network participation.
Disable Mechanism: A sentinel file can be placed to disable readiness checks, useful during maintenance or debugging.

Interaction with Other System Components

Heimdall Node: This script queries Heimdall node’s REST and RPC endpoints to determine synchronization and network status.
Container Orchestrator: The readiness script is designed to be integrated with orchestrators like Kubernetes, which use the exit codes to manage pod traffic routing.
Monitoring/Alerting: The readiness status can be used by monitoring tools to infer node health.
Peer Network: The peer count check ensures the node is part of the peer network before marking it ready.

Mermaid Flowchart Diagram

flowchart TD
    A[Start] --> B{Check if /root/disable_readiness exists}
    B -- Yes --> C[Print "readiness probe disabled"]
    C --> D[Exit 0 (Ready)]
    B -- No --> E[Fetch /syncing from localhost:1317]
    E -->|Success| F[Fetch /net_info from localhost:26657]
    E -->|Fail| G[Exit 1 (Not Ready)]
    F -->|Success| H[Fetch /status from localhost:26657]
    F -->|Fail| G
    H -->|Success| I[Parse JSON fields: syncing, catching_up, n_peers]
    H -->|Fail| G
    I --> J{Is syncing == false AND catching_up == false?}
    J -- No --> K[Print "heimdall is still syncing"]
    K --> G
    J -- Yes --> L{Is n_peers > 0?}
    L -- Yes --> M[Print "heimdall is synced with n_peers peers"]
    M --> D
    L -- No --> N[Print "heimdall is synced, but has no peers"]
    N --> G

Summary

The `readiness.sh` script is a simple yet critical utility that enables Kubernetes or other container orchestrators to determine when a Heimdall blockchain node is fully synchronized and network-connected, hence ready to serve traffic. It relies on querying local RPC endpoints and interpreting their JSON responses to make readiness decisions. The ability to disable the probe and the requirement of active peer connections enhance its flexibility and reliability in production environments.