Cluster Setup and IPFS

Purpose

This subtopic addresses the automated provisioning and configuration of Kubernetes clusters tailored for the ShapeShift Unchained platform, with a particular focus on integrating a decentralized file storage solution via IPFS (InterPlanetary File System). It solves the problem of managing scalable, reliable infrastructure environments and providing decentralized storage capabilities as a native part of the deployment process, thereby enabling blockchain services and APIs to leverage distributed storage seamlessly.

Functionality

The cluster setup leverages Pulumi scripts to:

• Provision Kubernetes Clusters: Using a dedicated EKSClusterLauncher utility, clusters are created with configurable parameters such as availability zones, autoscaling groups, network CIDR blocks, and node profiles. This ensures consistent and repeatable cluster environments optimized for blockchain workloads.

• Namespace Creation: Multiple Kubernetes namespaces are created dynamically, including a default namespace and environment-specific namespaces, to logically isolate services and resources.

• Docker Image Build & Push: If configured, the system automates building and pushing Docker images to a container registry, ensuring that the latest service images are available within the cluster.

• IPFS Cluster Deployment: An IPFS cluster is deployed within a dedicated namespace to provide decentralized, fault-tolerant file storage. This involves:

  • Creating necessary Kubernetes Secrets and ConfigMaps to manage sensitive data (cluster secrets) and configuration scripts.

  • Deploying two core container sets:

    • The ipfs daemon itself, responsible for peer-to-peer file storage and retrieval.

    • The ipfs-cluster service, managing cluster coordination, pinning, and replication.

  • Exposing services and ports related to swarm networking, APIs, and proxy functions to allow intra-cluster and external communication.

  • Configuring health checks (liveness probes) to monitor pod health.

  • Defining persistent volume claims for durable storage.

  • Setting up TLS certificates and Traefik ingress routing for secure, external access to the IPFS gateway.

This automation encapsulates infrastructure setup and decentralized storage provisioning into a single, easily repeatable deployment process.

Integration

Cluster Setup and IPFS integrates tightly with the parent topic of **Deployment Automation** by providing the foundational Kubernetes environment and storage infrastructure upon which blockchain node daemons, indexers, and API services run.

• The provisioned cluster forms the runtime environment for all other blockchain coinstacks and services.

• Namespaces created here segregate workloads, allowing coinstacks and supporting services to coexist without resource conflicts.

• The IPFS cluster complements blockchain services by enabling decentralized file storage, which can be utilized by components requiring persistent, distributed data storage beyond traditional block data—for example, storing large off-chain metadata or distributed artifacts referenced by blockchain transactions.

• By automating TLS certificate issuance and ingress routing, it aligns with the overall deployment strategy to provide secure, externally accessible services.

This subtopic adds a new dimension to deployment automation by integrating decentralized storage infrastructure natively, which is not covered by other subtopics focused solely on Kubernetes resource management or Docker image automation.

Diagram

flowchart TD
  A[Start Pulumi Deployment] --> B[Create EKS Kubernetes Cluster]
  B --> C[Create Namespaces (default & env-specific)]
  C --> D[Build & Push Docker Images (if configured)]
  D --> E[Deploy IPFS Cluster in 'unchained-infra' namespace]
  E --> F[Create Secrets & ConfigMaps for IPFS]
  F --> G[Deploy IPFS Daemon & IPFS-Cluster StatefulSet]
  G --> H[Set up Persistent Volumes]
  H --> I[Configure TLS Certs & Traefik IngressRoute]
  I --> J[IPFS Gateway Available with Secure Access]
  J --> K[Cluster Ready for Blockchain Services Deployment]

Code Snippet Highlights

• Cluster Creation with EKSClusterLauncher:

  const cluster = await EKSClusterLauncher.create(name, {
    allAZs: config.eks.allAZs,
    autoscaling: config.eks.autoscaling,
    cidrBlock: config.eks.cidrBlock,
    nodeGroups: config.eks.nodeGroups,
    region: config.eks.region,
    rootDomainName: config.rootDomainName,
    // additional config...
  })
  

• Dynamic Namespace Creation:

  namespaces.forEach(async (namespace) => {
    new core.v1.Namespace(namespace, { metadata: { name: namespace } }, { provider })
  })
  

• IPFS Deployment Invocation:

  deployIpfs({
    namespace: 'unchained-infra',
    domain: config.rootDomainName,
    provider,
  })
  

• IPFS StatefulSet with Multiple Containers and Persistent Storage:

  new k8s.apps.v1.StatefulSet(
    'ipfs',
    {
      spec: {
        replicas: 3,
        template: podSpec,
        volumeClaimTemplates: [
          { metadata: { name: 'cluster-storage' }, spec: { /* 5Gi gp3 storage */ } },
          { metadata: { name: 'ipfs-storage' }, spec: { /* 200Gi gp3 storage */ } },
        ],
      }
    },
    { provider }
  )
  

This encapsulates how the cluster is bootstrapped and how IPFS is integrated as a core infrastructure component to support decentralized file storage needs within the Kubernetes environment.