Project Overview

Project Purpose and Objectives

This project, ShapeShift Unchained, is a comprehensive blockchain infrastructure platform designed to provide reliable, scalable, and performant APIs and services for multiple blockchain networks. Its primary goal is to enable seamless interaction with diverse blockchain ecosystems by offering standardized APIs for account management, transaction history, sending transactions, fee estimation, and real-time event subscriptions.

Key objectives include:

• Multi-Blockchain Support: Support a wide range of blockchains including Bitcoin, Ethereum and its Layer 2 solutions (Arbitrum, Optimism), Cosmos-based chains (Thorchain, Thorchain-v1), Solana, and others, via modular coinstack implementations.

• High Availability and Scalability: Deploy blockchain node services, indexers, and APIs on Kubernetes clusters with automated health checks, readiness/liveness probes, and autoscaling support.

• Unified API Layer: Provide consistent RESTful and WebSocket APIs abstracting blockchain-specific complexities, enabling clients to interact with blockchain data and operations efficiently.

• Infrastructure as Code: Automate deployment and management of blockchain services, stateful sets, ingress routes, and Docker images using Pulumi.

• Observability and Monitoring: Integrate Prometheus and Grafana monitoring stacks with alerting, dashboards, and logging to ensure system health and performance visibility.

• Developer Tooling and Testing: Include performance testing scripts, Docker Compose setups, and local development configurations for ease of contribution and validation.

Major Functionalities and Implementation Details

• Coindstack Services: Each supported blockchain is implemented as a 'coinstack' with specialized services:

  • Daemon Nodes: Run blockchain node clients (e.g., geth, bitcoind, nethermind, thornode) managed by shell scripts for lifecycle, readiness, and liveness probes.

  • Indexer Services: Blockbook-based indexers providing efficient blockchain data queries and caching with readiness checks.

  • API Servers: Express.js (Node.js) or Go-based REST and WebSocket APIs exposing blockchain data and transaction operations.

  • Supporting Services: Some blockchains have additional components (e.g., Heimdall for Polygon, IPFS cluster for distributed file storage).

• Deployment Automation:

  • Pulumi TypeScript scripts define and deploy Kubernetes clusters, namespaces, services, StatefulSets, ConfigMaps, and IngressRoutes.

  • Docker images are built and pushed to registries with caching and tag management.

  • Kubernetes manifests include health probes implemented as shell scripts to monitor node synchronization and service readiness.

• API Layer:

  • Common API interfaces and models are defined to abstract blockchain-specific details.

  • EVM-compatible chains reuse common controllers and services with chain-specific configurations.

  • UTXO-based chains implement their own service and controller logic with Blockbook integration.

  • Solana and Cosmos chains have dedicated implementations handling their unique data models.

• Monitoring and Alerting:

  • Prometheus metrics are collected for HTTP requests and WebSocket connections.

  • Grafana dashboards visualize blockchain-specific metrics.

  • Alertmanager routes alerts to Discord with custom templates.

• Developer Utilities:

  • Shell scripts for Kubernetes probe checks and node initialization.

  • Docker Compose files for local development environments.

  • k6 load test scripts for benchmarking API performance.

  • Lerna monorepo setup managing Node.js packages.

Example Workflows and Use Cases

1. Deployment of a New Coinstack (e.g., Litecoin)

• Configuration Preparation: Developers modify the sample.env and config.json files within the Litecoin coinstack directory to specify environment variables and service parameters.

• Pulumi Deployment: Run the Pulumi deployment script (node/coinstacks/litecoin/pulumi/index.ts) which:

  • Reads configuration and environment files.

  • Constructs service arguments for daemon and indexer services.

  • Calls deployCoinstack to create Kubernetes resources like StatefulSets, Services, ConfigMaps, and ingress routes.

• Service Startup and Health Checks:

  • Daemon and indexer containers start with lifecycle scripts managing startup, readiness, and liveness probes (e.g., readiness.sh, liveness.sh).

  • Kubernetes monitors pod health and restarts services if probes fail.

• API Interaction:

  • The Litecoin API server (api/src/app.ts) listens on port 3000, serving REST endpoints and WebSocket connections.

  • Clients can query account info, transaction history, send transactions, and subscribe to blockchain events.

• Monitoring:

  • Prometheus collects metrics from the API and node services.

  • Grafana dashboards visualize metrics with alerting configured for anomalies.

2. Local Development Setup with Docker Compose

• Use provided docker-compose.yml files for various coinstacks (e.g., node/coinstacks/litecoin/docker-compose.yml) to spin up local containers for the daemon, indexer, and API services.

• Environment variables are sourced from .env files.

• Traefik reverse proxy routes API requests to the appropriate containers.

• Developers can run API servers locally with hot reload using nodemon.

• Use shell scripts such as readiness.sh to simulate Kubernetes probes locally.

3. Real-time Blockchain Event Subscription

• API servers establish WebSocket connections to the Blockbook indexer.

• Clients connect to the API WebSocket endpoint to subscribe to transaction addresses.

• The Registry class manages subscriptions and dispatches transaction events to clients.

• Transaction handlers extract involved addresses and broadcast relevant updates.

Stack and Technologies

Core Technologies

• Node.js & TypeScript: Primary language and runtime for API servers and infrastructure scripts.

• Go: Used for Thorchain coinstack services and API implementations.

• Kubernetes: Orchestrates containerized blockchain node services, indexers, and APIs.

• Pulumi: Infrastructure as code tool for Kubernetes cluster creation, resource provisioning, and Docker image management.

• Docker: Containerization platform for local development and deployment.

• Express.js: Web framework for RESTful API servers.

• WebSocket: For real-time event subscriptions.

• Blockbook: Blockchain indexer providing efficient querying of blockchain data.

• Prometheus & Grafana: Monitoring and visualization stack with alerting support.

• Traefik: Reverse proxy for routing HTTP and WebSocket traffic.

Key Libraries and Frameworks

• @shapeshiftoss/blockbook: Blockchain indexer client and WebSocket integration.

• @shapeshiftoss/common-api: Common API interfaces, middleware, and utilities.

• @shapeshiftoss/logger: Structured logging utility.

• @shapeshiftoss/websocket: WebSocket client and connection management.

• viem: Ethereum and EVM-compatible blockchain client utilities.

• axios and axios-retry: HTTP client with retry logic.

• tsoa: OpenAPI generation and API route decorators for TypeScript.

• typescript, nodemon: Development tooling for TypeScript projects.

• jq, curl: Shell utilities for JSON parsing and HTTP requests used in probe scripts.

• helm (implied via Pulumi): Kubernetes package management.

Rationale

• Node.js + TypeScript provides rapid development with strong typing and ecosystem support for blockchain integrations.

• Go is used for high-performance blockchain services (Thorchain) that benefit from concurrency and static compilation.

• Kubernetes + Pulumi supports declarative, scalable, and repeatable deployment workflows.

• Blockbook standardizes blockchain data indexing across UTXO chains for efficient querying.

• Prometheus/Grafana enable observability essential for production readiness.

• WebSocket enables efficient real-time data delivery for blockchain events.

High-Level Architecture

The system architecture is modular and layered, supporting multiple blockchain coinstacks deployed as Kubernetes StatefulSets and services. The key components and their interactions are:

• Frontend (Client Applications)

  • Connect to REST API endpoints for blockchain data retrieval.

  • Connect to WebSocket endpoints for real-time transaction subscriptions.

• API Layer (Node.js / Go Servers)

  • Expose REST and WebSocket APIs for each blockchain.

  • Manage connections to the blockchain indexers and daemons.

  • Implement blockchain-specific logic and data formatting.

  • Use shared common APIs and middleware for consistency.

• Indexer Layer (Blockbook)

  • Index blockchain data for fast queries.

  • Provide WebSocket streams for new blocks and transactions.

  • Indexed data served to API layer.

• Blockchain Node Layer (Daemons)

  • Run blockchain full nodes or clients per supported blockchain.

  • Maintain blockchain state and relay data to indexers.

  • Managed with shell scripts for lifecycle and health checks.

• Infrastructure Layer

  • Kubernetes cluster hosting all services with namespaces per environment.

  • Pulumi manages cluster creation, namespaces, deployments, services, ingress, and Docker image building/pushing.

  • Traefik reverse proxy routes external traffic to internal services.

  • Monitoring stack (Prometheus, Grafana, Alertmanager) deployed for observability.

graph TB
  Client[Client Applications] -->|REST API| API[API Servers]
  Client -->|WebSocket| API
  API -->|Query| Indexer[Blockbook Indexer]
  API -->|RPC Calls| Daemon[Blockchain Daemon Nodes]
  Indexer --> Daemon
  Daemon --> Blockchain[Blockchain Networks]
  API -->|Metrics| Prometheus[Prometheus Monitoring]
  Prometheus --> Grafana[Grafana Dashboards]
  API --> Traefik[Traefik Reverse Proxy]
  Traefik --> Client

Developer Navigation

Frontend Developers Start Here

• This project does not include frontend UI code but exposes REST and WebSocket APIs.

• Explore node/coinstacks/*/api/src/app.ts for server setup and routing.

• Review API controllers in node/coinstacks/*/api/src/controller.ts for endpoint logic.

• Use Swagger specs (*.json files) for API contracts per coinstack.

Backend Developers Focus on:

• Coinstack API Servers: Located under node/coinstacks/{coin}/api/src/

• Daemon Management: Lifecycle scripts in node/coinstacks/{coin}/daemon/ with probes in shell scripts.

• Indexer Configuration: Blockbook indexer config under node/coinstacks/{coin}/indexer/

• Pulumi Deployment: Infrastructure as code in node/coinstacks/{coin}/pulumi/index.ts

• Common API Logic: Shared business logic in node/coinstacks/common/api/src/

• Blockchain-Specific Implementation: Go coinstacks under go/coinstacks/thorchain/

DevOps / Infrastructure

• Cluster Setup: Pulumi cluster deployment in pulumi/src/cluster/index.ts

• Docker Images: Build and push utilities in pulumi/src/docker.ts and pulumi/src/cluster/docker.ts

• Stateful Service Deployment: Kubernetes StatefulSets configured in pulumi/src/statefulService.ts

• Monitoring Stack: Prometheus, Grafana, Alertmanager setup under monitoring/src/

• Deployment Scripts: Shell scripts for probes in node/scripts/ and coinstack daemon scripts.

Testing and Performance

• Load Testing: K6 scripts under k6/ for Bitcoin and Ethereum performance testing.

• Docker Compose: Local development environments using docker-compose.yml files in coinstack directories.

• Logging: Loki and Grafana integration documented in docs/tooling.md.

Visual Diagrams

Component Diagram - High-Level System Architecture

graph TB
  Client[Client Application] -->|HTTP REST / WS| API[API Servers]
  API -->|Query| Indexer[Blockbook Indexer]
  API -->|RPC Calls| Daemon[Blockchain Node Daemons]
  Indexer --> Daemon
  Daemon --> Blockchain[Blockchain Networks]
  API -->|Metrics| Prometheus[Prometheus Metrics]
  Prometheus --> Grafana[Grafana Dashboard]
  API --> Traefik[Traefik Proxy]
  Traefik --> Client

Flowchart - Example Workflow: API Request Handling for Transaction History

flowchart TD
  ClientRequest[Client sends Tx History Request] --> APIServer[API Server]
  APIServer --> Validate[Validate Input & Auth]
  Validate --> QueryIndexer[Query Blockbook Indexer]
  QueryIndexer --> FetchTxs[Fetch Transactions]
  FetchTxs --> Format[Format & Enrich Data]
  Format --> ReturnResponse[Return JSON Response]
  ReturnResponse --> ClientRequest

This overview provides a technical roadmap for developers to understand the project goals, core components, and workflows. It highlights the modular coinstack architecture, deployment automation, API services, and monitoring infrastructure. Developers can use this as a guide to navigate the codebase, contribute, and maintain the system effectively.