Infrastructure Scripts

Infrastructure Scripts in this project encompass shell scripts and utilities that support the operational lifecycle of blockchain node daemons and related Kubernetes services. These scripts are primarily focused on three key areas: Kubernetes probe checks, daemon lifecycle management, and snapshot/restore operations. They provide essential automation and health monitoring capabilities to ensure the reliability, readiness, and consistency of blockchain nodes deployed within Kubernetes environments.

Overview

Infrastructure scripts serve as foundational tools for the smooth running of blockchain node services (daemons) and their integration into Kubernetes-managed clusters. They address the following problems:

• Health and readiness monitoring: Kubernetes requires liveness and readiness probes to determine if a pod is functioning correctly or ready to serve traffic. Custom shell scripts implement these probes tailored to blockchain node specifics.

• Daemon initialization and configuration: Blockchain nodes require bootstrapping steps such as chain initialization, genesis file setup, and peer configuration before starting.

• Snapshot management: To speed up node synchronization, snapshot download and extraction automate blockchain state restoration from trusted sources.

• Dynamic environment configuration: Scripts parse external metadata to derive configuration variables dynamically, supporting flexible deployment across different blockchain networks.

These scripts abstract away manual operational steps and embed best practices for stateful blockchain node management on Kubernetes.

Key Functionalities and Workflows

1. Kubernetes Probe Scripts

The probe scripts implement monitoring hooks designed for Kubernetes liveness, readiness, and startup probes. These scripts typically execute simple status checks or wait indefinitely to keep a container alive during startup.

• Example: probe.sh

  This script is a minimal "dummy" probe that keeps the container alive by tailing an empty stream indefinitely and correctly handles termination signals:

  #!/bin/sh
  
  set -e
  
  trap 'exit' SIGTERM SIGINT
  
  tail -f /dev/null &
  
  wait $!
  

  This kind of script is used as a placeholder or baseline liveness probe ensuring the container does not exit unexpectedly, allowing Kubernetes to monitor pod health without complex logic.

• In other coinstacks, more specialized probe scripts exist to verify blockchain node sync status, API responsiveness, or indexer health. These scripts typically use CLI commands or HTTP queries to confirm the node is fully operational.

2. Daemon Lifecycle and Initialization (run.sh)

The core daemon bootstrap script (`go/scripts/run.sh`) manages environment configuration, dynamic chain metadata loading, snapshot restoration, and node initialization before launching the blockchain binary.

**Workflow highlights:**

• Dynamic Chain Metadata Loading:

  If an environment variable CHAIN_JSON is provided, the script fetches chain metadata from the specified URL (usually a JSON descriptor of chain parameters). It extracts important configuration such as:

  • Chain ID

  • Peer seeds and persistent peers for P2P connections

  • Genesis file URL

  • Daemon binary name and home directory

  This dynamic loading allows the same script to be reused across different blockchain networks by simply pointing to different metadata sources.

• Snapshot Quick Sync:

  To accelerate node synchronization, the script supports downloading and extracting pre-built snapshots. This process includes:

  • Querying a snapshot index URL (provided by SNAPSHOT_QUICKSYNC) for matching snapshot files based on chain ID and pruning strategy.

  • Determining the compression format (e.g., tar.gz, tar.lz4, tar.zst) dynamically from the snapshot URL.

  • Downloading the snapshot with progress visualization (using pv) and extracting it accordingly.

  • Moving extracted data and WASM files into their appropriate directories.

• Genesis File Download and Initialization:

  If the node configuration directory ($CONFIG_PATH) does not exist, the script initializes the blockchain node with the configured moniker and chain ID by invoking the daemon binary's init command. It then downloads the genesis file, handles decompression if necessary, and places it in the config directory.

• Peer Configuration:

  The script sets the P2P networking peers by overriding the config.toml file with the seeds and persistent peers from the environment or metadata. This ensures the node connects to the correct peers on startup.

• Priv Validator State Pre-seeding:

  To avoid known issues with missing priv_validator_state files, the script pre-seeds a default JSON file in the node data directory if it does not exist.

• Polkachu Seed Node Integration:

  When enabled via environment variables, the script fetches and appends Polkachu seed nodes to the peer configuration dynamically, facilitating P2P connectivity enhancements.

• Debugging Support:

  Environment variables (DEBUG) enable verbose output and environment inspection to aid troubleshooting.

• Final Execution:

  The script ends by executing the daemon binary with any provided arguments, effectively launching the blockchain node process.

**Excerpt illustrating dynamic chain metadata loading and snapshot handling:**

if [ -n "$CHAIN_JSON" ]; then
  CHAIN_METADATA=$(curl -s $CHAIN_JSON)

  export CHAIN_ID="${CHAIN_ID:-$(echo $CHAIN_METADATA | jq -r .chain_id)}"
  export P2P_SEEDS="${P2P_SEEDS:-$(echo $CHAIN_METADATA | jq -r '.peers.seeds | map(.id+"@"+.address) | join(",")')}"
  export P2P_PERSISTENT_PEERS="${P2P_PERSISTENT_PEERS:-$(echo $CHAIN_METADATA | jq -r '.peers.persistent_peers | map(.id+"@"+.address) | join(",")')}"
  export GENESIS_URL="${GENESIS_URL:-$(echo $CHAIN_METADATA | jq -r '.codebase.genesis.genesis_url? // .genesis.genesis_url? // .genesis?')}"
  export PROJECT_BIN="${PROJECT_BIN:-$(echo $CHAIN_METADATA | jq -r '.codebase.daemon_name? // .daemon_name?')}"
  export PROJECT_DIR="${PROJECT_DIR:-$(echo $CHAIN_METADATA | jq -r '.codebase.node_home? // .node_home?')}"
fi

# Snapshot download and extraction
if [[ -n $SNAPSHOT_QUICKSYNC && ! -f "$PROJECT_ROOT/data/priv_validator_state.json" ]]; then
  SNAPSHOT_PRUNING="${SNAPSHOT_PRUNING:-default}"
  SNAPSHOT_URL=`curl -s $SNAPSHOT_QUICKSYNC | jq -r --arg FILE "$CHAIN_ID-$SNAPSHOT_PRUNING"  'first(.[] | select(.file==$FILE)) | .url'`
  
  # ... determine snapshot format and extract accordingly ...
fi

Interaction with Other System Components

• Kubernetes Deployment:

  These scripts are embedded in container images for blockchain daemons and configured as the container entrypoint or lifecycle hooks in Kubernetes StatefulSets. For example, the readiness and liveness probes configured in Kubernetes Pod specs invoke these scripts or their variants to check node health.

• Pulumi Deployment Automation:

  The Pulumi infrastructure scripts reference these shell scripts when defining StatefulSets and container specs. This ensures that lifecycle and probe scripts are bundled with the containers and that Kubernetes can perform health checks effectively.

• Daemon and Indexer Services:

  The snapshot and initialization workflows orchestrated by these scripts ensure that daemon nodes start in a consistent state, which is essential for indexers (e.g., Blockbook) that rely on daemon node data to build blockchain indices.

• Logging and Monitoring:

  While these scripts do not directly handle logging or telemetry, their proper execution ensures that the blockchain nodes remain healthy and synchronized, which in turn affects metrics collected by Prometheus and alerts visualized in Grafana.

Important Concepts and Design Patterns

• Dynamic Configuration via Metadata:

  Using external JSON metadata (CHAIN_JSON) to derive runtime configuration parameters reduces duplication and increases flexibility, allowing a single script to support multiple blockchain deployments without modification.

• Idempotent Initialization:

  The scripts check for existing configuration and data directories before performing initialization steps, ensuring safe restarts and upgrades without data loss or reinitialization errors.

• Snapshot-Based Fast Sync:

  Incorporating snapshot download and extraction into the startup process optimizes node synchronization time dramatically, which is critical for large blockchain datasets.

• Signal Handling and Container Lifecycle Awareness:

  Probe scripts handle termination signals gracefully, ensuring Kubernetes can manage pod lifecycle correctly without premature container exits.

• Use of Standard Unix Tools:

  The scripts leverage common Unix utilities (curl, jq, tar, pv, sed) to achieve complex tasks with minimal dependencies, enhancing portability and ease of debugging.

Mermaid Diagram: Flowchart of Daemon Initialization and Startup Process

flowchart TD
  Start[Start Container / Script] --> LoadMetadata{Is CHAIN_JSON set?}
  LoadMetadata -- Yes --> FetchMetadata[Fetch Chain Metadata JSON]
  FetchMetadata --> ExtractConfig[Extract Chain ID, Peers, Genesis URL, etc.]
  LoadMetadata -- No --> SkipMetadata[Skip Metadata Loading]

  ExtractConfig --> CheckConfigDir{Config directory exists?}
  SkipMetadata --> CheckConfigDir

  CheckConfigDir -- No --> InitChain[Run daemon init with moniker and chain ID]
  InitChain --> DownloadGenesis{Is GENESIS_URL set?}
  CheckConfigDir -- Yes --> SkipInit[Skip Chain Initialization]
  SkipInit --> DownloadGenesis

  DownloadGenesis -- Yes --> DownloadAndExtract[Download & extract genesis file]
  DownloadGenesis -- No --> SkipGenesis[Skip Genesis Download]

  DownloadAndExtract --> ConfigurePeers[Set seeds and persistent peers in config.toml]
  SkipGenesis --> ConfigurePeers

  ConfigurePeers --> CheckSnapshot{Is SNAPSHOT_QUICKSYNC set?}
  CheckSnapshot -- Yes --> DownloadSnapshot[Download and extract snapshot]
  CheckSnapshot -- No --> SkipSnapshot[Skip Snapshot]

  DownloadSnapshot --> PreseedPrivVal[Preseed priv_validator_state.json if missing]
  SkipSnapshot --> PreseedPrivVal

  PreseedPrivVal --> ExecuteDaemon[Execute daemon binary with arguments]
  ExecuteDaemon --> End[Daemon Running]

This documentation outlines the purpose, workflow, and integration of infrastructure scripts within the system. These scripts are vital for ensuring blockchain daemons start reliably, remain healthy under Kubernetes orchestration, and can sync efficiently using snapshots. Their design embraces flexibility, idempotency, and container lifecycle awareness, enabling robust multi-chain deployments.