Deployment Automation

Overview

The **Deployment Automation** module provides an infrastructure-as-code solution that automates the deployment of blockchain node daemons, indexers, and API services onto Kubernetes clusters. This automation is designed to efficiently provision and configure all essential blockchain coinstack components while managing dependencies, service lifecycles, networking, and observability integrations.

The module leverages Pulumi, a programmable infrastructure tool, to declaratively define Kubernetes resources such as StatefulSets, Deployments, Services, ConfigMaps, IngressRoutes, and Certificates, enabling reproducible, scalable, and maintainable deployments. This system abstracts the complexities of manual Kubernetes configuration and ensures consistent environments across blockchain coinstacks.

Core Concepts and Purpose

• Automated Kubernetes Resource Management: Define and deploy StatefulSets for blockchain daemons and indexers, Deployments for APIs, Services for network access, and ingress routing with TLS certificates.

• Dynamic Configuration Injection: Inject environment variables, configuration files, and lifecycle scripts (init, startup, readiness, liveness probes) into containers via ConfigMaps and Secrets.

• Docker Image Build and Push Automation: Build container images for APIs and optionally push them to Docker registries with caching and tagging based on the coinstack's source and configuration hash.

• Modular Coinstack Support: Deploy multiple blockchain coinstacks (e.g., Bitcoin, Litecoin, Avalanche) with customizable service parameters and health checks.

• Integration with Monitoring and TLS: Automatically create Prometheus ServiceMonitors and cert-manager Certificates for observability and secure ingress.

This module solves the problem of complex, error-prone manual Kubernetes deployments by providing a unified, extensible, and programmable way to deploy blockchain infrastructure consistent with operational best practices.

How Deployment Automation Works

Entry Point: Coinstack Deployment

Each blockchain coinstack defines a Pulumi entry script (e.g., `node/coinstacks/litecoin/pulumi/index.ts`) that:

1. Reads environment samples (sample.env) and configuration files.

2. Retrieves Kubernetes access credentials and namespace.

3. Constructs service arguments for each coinstack service (daemon, indexer, etc.) with ports, environment variables, probes, and config maps.

4. Calls the core method deployCoinstack with all relevant parameters.

Example snippet from Litecoin coinstack deployment script:

const coinServiceArgs = config.statefulService?.services?.map((service): CoinServiceArgs => {
  switch (service.name) {
    case 'daemon':
      return {
        ...service,
        ports: { 'daemon-rpc': { port: 8332 } },
        readinessProbe: { periodSeconds: 30, failureThreshold: 10 },
      }
    case 'indexer':
      return {
        ...service,
        ...defaultBlockbookServiceArgs,
        configMapData: { 'indexer-config.json': readFileSync('../indexer/config.json').toString() },
      }
    default:
      throw new Error(`no support for coin service: ${service.name}`)
  }
})

return deployCoinstack({
  appName,
  coinServiceArgs,
  coinstack,
  coinstackType: 'node',
  config,
  kubeconfig,
  namespace,
  sampleEnv,
})

Deploying a Coinstack (deployCoinstack)

The `deployCoinstack` function (in `pulumi/src/coinstack.ts`) orchestrates deployment by:

• Creating a Kubernetes Secret from environment variables parsed from the sample .env file and process environment for sensitive data.

• Deploying the API service by invoking deployApi.

• Creating StatefulSets for coin services such as daemons and indexers by calling deployStatefulService.

This approach centralizes coinstack deployment logic and enforces consistent naming, labeling, and configuration patterns.

Key excerpt from `deployCoinstack`:

const secretData = getSecretData(sampleEnv)
new k8s.core.v1.Secret(assetName, { metadata: { name: assetName, namespace }, stringData: secretData }, { provider })

await deployApi({ ...args, assetName, baseImageName, provider })

const coinServices = (coinServiceArgs ?? []).map((_args) => createCoinService(_args, assetName))

await deployStatefulService(appName, assetName, provider, namespace, config, coinServices, volumes)

Defining Stateful Services (createCoinService and deployStatefulService)

• createCoinService constructs a service specification object representing a coinstack component (daemon, indexer, etc.). It reads lifecycle shell scripts (init.sh, startup.sh, liveness.sh, readiness.sh) from the service directories and includes them as ConfigMap data and container probes. It also sets resource requests/limits, environment variables, ports, volume mounts, and security context.

• deployStatefulService takes an array of these service specifications and creates Kubernetes StatefulSets and Services. It merges all ConfigMaps, volumes, and container specs for the StatefulSet Pod template. In addition, it configures:

  • Service selectors and ports.

  • TLS certificates using cert-manager.

  • Traefik ingress routing with path prefix stripping middleware for fine-grained HTTP routing.

  • Rolling update strategies and parallel pod management.

  • PersistentVolumeClaims for storage with optional AWS EBS-specific storage parameters.

Excerpt showing container and volume specification:

const serviceContainer: k8s.types.input.core.v1.Container = {
  name,
  image: args.image,
  command: initScript && !args.command ? ['/init.sh'] : args.command,
  env,
  ports: ports.map(({ port: containerPort, name }) => ({ containerPort, name })),
  startupProbe: startupProbe && { exec: { command: ['/startup.sh'] } },
  livenessProbe: livenessProbe && { exec: { command: ['/liveness.sh'] } },
  readinessProbe: readinessProbe && { exec: { command: ['/readiness.sh'] } },
  volumeMounts: [
    { name: `data-${args.name}`, mountPath: args.dataDir ?? '/data' },
    { name: 'config-map', mountPath: '/init.sh', subPath: `${args.name}-init.sh` },
    // ... other scripts
  ],
}

API Service Deployment (deployApi)

The API services are deployed separately as Kubernetes Deployments (stateless), managed by `deployApi` in `pulumi/src/api.ts`. This function:

• Builds and pushes Docker images for the API based on the coinstack type (Node.js or Go), using a tagging mechanism derived from the source hash to ensure immutability and cache efficiency.

• Creates a ClusterIP Service exposing the API.

• Sets up Prometheus ServiceMonitors for metrics scraping.

• Creates TLS certificates and Traefik IngressRoutes for secure HTTPS access with domain naming conventions based on environment and asset.

• Supports optional horizontal pod autoscaling based on CPU utilization thresholds.

• Specifies container readiness and liveness probes using HTTP health endpoints.

Excerpt illustrating image building and Deployment creation:

const tag = await getCoinstackHash(coinstack, buildArgs, coinstackType)
const imageName = config.dockerhub ? `${config.dockerhub.username}/${repositoryName}:${tag}` : `shapeshiftdao/${repositoryName}:${tag}`

// Build and push image if needed (not shown)

const apiDeployment = new k8s.apps.v1.Deployment(
  name,
  {
    spec: {
      replicas: config.api.replicas,
      template: {
        spec: {
          containers: [{
            name: tier,
            image: imageName,
            ports: [{ containerPort: port, name: 'http' }],
            readinessProbe: { httpGet: { path: '/health', port } },
            livenessProbe: { httpGet: { path: '/health', port } },
            // ...
          }],
        },
      },
    },
  },
  { provider, dependsOn: deployDependencies }
)

Interaction with Other System Parts

• Coinstack Service Definitions: The deployment scripts in each coinstack (node/coinstacks/{coin}/pulumi/index.ts) prepare custom service configurations (ports, environment variables, config files, probes) and invoke the deployment automation core.

• Shell Scripts for Probes: Lifecycle and health check scripts (init.sh, liveness.sh, readiness.sh, startup.sh) are read by createCoinService and included in ConfigMaps, mounted into containers, and configured as Kubernetes probes to monitor service health.

• Docker Image Automation: The API deployment includes logic to build, tag, and push Docker images dynamically based on source code and configuration hashes, enabling continuous integration and consistent deployment versions.

• Ingress and TLS: The module configures Traefik ingress routes and cert-manager TLS certificates for all services, enabling secure and routable access to deployed blockchain nodes and APIs.

• Monitoring Integration: Prometheus ServiceMonitors are automatically created for APIs and StatefulSets to facilitate metrics collection and alerting.

• Persistent Storage: Stateful services use PersistentVolumeClaims with configurable storage classes and AWS-specific performance tuning (IOPS, throughput) annotations.

Important Concepts and Design Patterns

• Infrastructure as Code (IaC): The entire deployment is defined programmatically using Pulumi, allowing reuse, versioning, and automation.

• Modular Service Abstraction: Services are described declaratively using CoinServiceArgs and assembled into StatefulSets or Deployments, promoting modularity.

• Dynamic ConfigMap Injection: Lifecycle scripts and configuration files are dynamically loaded and injected as ConfigMaps, decoupling runtime behavior from container images.

• Health Probes as Executable Scripts: Instead of static HTTP checks alone, custom shell scripts are used for readiness, liveness, and startup probes, tailored to blockchain node synchronization states.

• Automated Image Tagging and Caching: Docker images are tagged based on content hashes, enabling efficient rebuilds and ensuring deployments use immutable images.

• Kubernetes Native TLS and Routing: TLS certificates and ingress routes are managed using cert-manager and Traefik CRDs, enabling zero-downtime secure access.

• Horizontal Pod Autoscaling: Optional CPU-based autoscaling is configured for API deployments, enhancing scalability under load.

Visual Diagram: Deployment Automation Workflow

flowchart TD
  Start[Start Deployment] --> ReadConfig[Read Env & Config Files]
  ReadConfig --> PrepareServices[Prepare Coin Services Args]
  PrepareServices --> DeploySecret[Create Kubernetes Secret]
  DeploySecret --> DeployAPI[Deploy API Deployment & Service]
  DeployAPI --> DeployStateful[Deploy StatefulSets for Daemon & Indexer]
  DeployStateful --> CreateIngress[Configure Ingress & TLS Certificates]
  CreateIngress --> SetupMonitoring[Setup Prometheus ServiceMonitors]
  SetupMonitoring --> End[Deployment Complete]

Summary

The **Deployment Automation** module is the backbone of the ShapeShift Unchained infrastructure setup, enabling teams to deploy blockchain coinstacks reliably and consistently on Kubernetes. Through Pulumi, it abstracts complex Kubernetes resource management into programmable, reusable functions that handle service lifecycle scripts, health probes, image management, ingress routing, and monitoring integration. It supports multiple coinstacks by allowing custom configurations per blockchain service, ensuring the operational health and secure accessibility of blockchain nodes and APIs.