liveness.sh

Overview

`liveness.sh` is a Bash script designed to serve as a **liveness probe** for an Optimism node (op-node), a component in a blockchain layer-2 scaling solution. The script verifies whether the node is actively processing new blocks by querying its synchronization status through a JSON-RPC endpoint and comparing the current block numbers to previously recorded values. If the node is progressing, the probe signals success; otherwise, it signals failure, indicating a potential stall.

The script also supports a manual override to disable the liveness check by the presence of a specific file.

Detailed Explanation

Purpose and Functionality

• Primary Goal: Detect if the op-node is live and syncing properly by checking its L1 and L2 block numbers.

• Mechanism:

  1. Queries the node’s JSON-RPC endpoint for sync status.

  2. Extracts current L1 and L2 block numbers.

  3. Compares these with previously saved block numbers.

  4. If both current block numbers have increased, the node is considered live.

  5. Otherwise, it reports the node as stalled.

• Disabling Probe: The presence of /data/disable_liveness disables the probe and causes it to exit successfully immediately.

Variables and Files

Step-by-Step Workflow

1. Check for Liveness Disable File

   • If /data/disable_liveness exists, the script prints "liveness probe disabled" and exits with status 0 (success).

2. Fetch Synchronization Status

   • Makes a JSON-RPC POST request to http://localhost:9545 with method optimism_syncStatus.

   • Fails silently (-sf) and exits with status 1 if the request fails.

3. Parse Block Numbers

   • Extracts current_l1.number and unsafe_l2.number from the JSON response using jq.

4. Initial Setup

   • If /data/.block_number does not exist, it creates the file with current block numbers and exits with status 1 (failure), indicating the probe is not yet ready.

5. Compare Current and Previous Block Numbers

   • Reads previous block numbers from /data/.block_number.

   • Overwrites the file with current block numbers.

   • If both current L1 and L2 block numbers are strictly greater than previous ones, prints "op-node is running" and exits 0 (success).

   • Otherwise, prints "op-node is stalled" and exits 1 (failure).

Important Implementation Details

• JSON-RPC Request:
  The script uses curl to POST a JSON-RPC request with method "optimism_syncStatus" to the node's HTTP endpoint on port 9545, expecting a structured JSON response with syncing information.

• Use of jq:
  JSON parsing is done via jq to extract numeric block numbers safely.

• File-based State Persistence:
  The script stores the last seen block numbers in a hidden file /.block_number to compare progress between invocations.

• Exit Codes for Liveness Probes:

  • 0 indicates success (node is live or probe disabled).

  • 1 indicates failure (node stalled or initial setup).

• Fail-fast on Request Failure:
  If the JSON-RPC call fails, the script exits immediately with failure to signal a liveness problem.

Usage Example

This script is typically used in Kubernetes or similar container orchestration environments as a **liveness probe** command.

livenessProbe:
  exec:
    command:
      - /bin/bash
      - /path/to/liveness.sh
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

• The orchestration system will periodically run this script.

• If it exits with status 0, the node is considered healthy.

• If it exits with status 1, the node is considered unhealthy and may be restarted.

Interaction with Other System Components

• Optimism Node (op-node):
  The script depends on the op-node exposing a JSON-RPC HTTP endpoint on localhost port 9545 with the method optimism_syncStatus.

• Persistent Data Storage:
  The file /data/.block_number must be persisted across probe executions to maintain state.

• Disable Mechanism:
  Presence of /data/disable_liveness allows manual disabling of the probe, likely managed externally (e.g., administrator intervention or automated maintenance).

Mermaid Flowchart Diagram

flowchart TD
    Start([Start])
    CheckDisable{File "/data/disable_liveness" exists?}
    ExitDisabled["Print \"liveness probe disabled\"\nExit 0"]
    FetchSyncStatus["Fetch sync status via JSON-RPC\n(curl POST to localhost:9545)"]
    CurlFail{"curl request success?"}
    ParseBlocks["Extract current L1 and L2 block numbers\nusing jq"]
    CheckFileExists{File "/data/.block_number" exists?}
    WriteFileFirstTime["Write current block numbers to file\nExit 1 (initial setup)"]
    ReadPrevBlocks["Read previous block numbers from file"]
    WriteCurrentBlocks["Overwrite file with current block numbers"]
    CompareBlocks{"Current L1 & L2 > Previous L1 & L2?"}
    ExitSuccess["Print \"op-node is running\"\nExit 0"]
    ExitStalled["Print \"op-node is stalled\"\nExit 1"]

    Start --> CheckDisable
    CheckDisable -- Yes --> ExitDisabled
    CheckDisable -- No --> FetchSyncStatus
    FetchSyncStatus --> CurlFail
    CurlFail -- No --> ExitStalled
    CurlFail -- Yes --> ParseBlocks
    ParseBlocks --> CheckFileExists
    CheckFileExists -- No --> WriteFileFirstTime
    CheckFileExists -- Yes --> ReadPrevBlocks
    ReadPrevBlocks --> WriteCurrentBlocks
    WriteCurrentBlocks --> CompareBlocks
    CompareBlocks -- Yes --> ExitSuccess
    CompareBlocks -- No --> ExitStalled

Summary

`liveness.sh` is a lightweight, robust liveness probe script for monitoring the health of an Optimism blockchain node by verifying block synchronization progress. It uses a combination of JSON-RPC querying, JSON parsing, and persistent state comparison to accurately determine if the node is operating correctly or stalled. The script's design allows easy integration with container orchestration liveness checks and supports manual disabling when needed.