config.yaml

Overview

`config.yaml` is a configuration file for **Prometheus Alertmanager**, tailored for the ShapeShift Unchained platform. Its primary purpose is to define how alerts generated by Prometheus are grouped, inhibited, routed, and delivered to various notification channels. This file focuses on managing alert routing logic, inhibition rules to reduce noise, and specifying Discord webhook receivers for sending formatted alert notifications.

The configuration ensures that alerts of different severities (`critical`, `warning`, `info`) and environments (`production` vs `development`) are handled appropriately to enable effective incident response. It also uses templates for consistent, readable alert messages in Discord.

Detailed Explanation of Configuration Sections

This file consists of several key sections:

1. global

• Purpose: Defines global settings applied across all alerts.

• Key Parameter:

  • resolve_timeout: 30m
    This sets the timeout duration (30 minutes) after which alerts are automatically resolved if no longer firing.

2. inhibit_rules

• Purpose: Defines rules to suppress (inhibit) lower severity alerts when higher severity alerts for the same issue are active, reducing alert noise and avoiding redundant notifications.

• Structure:
  Each inhibition rule has:

  • source_matchers: Criteria for the alert that triggers inhibition (usually higher severity)

  • target_matchers: Criteria for alerts to be inhibited (usually lower severity)

  • equal: List of label keys that must match between source and target alerts for inhibition to apply.

• Example Rule:

- source_matchers:
    - "severity = critical"
  target_matchers:
    - "severity =~ warning|info"
  equal:
    - "namespace"
    - "alertname"

This means: If a critical alert is firing for a given `namespace` and `alertname`, then all warning or info alerts with the same `namespace` and `alertname` are inhibited.

• Additional Details:

  • One special rule inhibits info alerts based on an alert named InfoInhibitor.

  • Another inhibits alerts relating to Kubernetes StatefulSet or Deployment mismatches when corresponding down alerts are firing.

3. route

• Purpose: Defines the global routing tree for alerts, specifying how alerts are grouped, matched, and forwarded to receivers.

• Key Parameters:

  • group_by: Labels by which alerts are grouped (e.g., alertname, namespace).

  • group_wait: Time to wait before sending the first notification of a group.

  • group_interval: Time to wait between notifications for new alerts in the same group.

  • repeat_interval: Interval before a notification for a resolved alert can be repeated.

  • receiver: Default receiver for alerts that don't match any sub-route.

  • routes: List of nested routing rules for specific alert matchers.

• Routing Rules:

Each route specifies:

• receiver: The Discord webhook to send matching alerts.

• group_by: Grouping labels tailored to the alert type.

• group_wait, group_interval, repeat_interval: Timing parameters adjusted per alert severity and environment.

• matchers: Label match expressions to filter alerts.

• Example Route:

- receiver: "discord_critical"
  group_by: ["alertname", "namespace", "statefulset"]
  group_wait: 5m
  group_interval: 30m
  repeat_interval: 1h
  matchers:
    - alertname = "UnchainedStatefulSetDown"
    - namespace = "unchained"
    - severity = "critical"

This routes critical statefulset down alerts in the `unchained` namespace to the `discord_critical` receiver.

• Environment Separation:
  Routes are defined separately for production (unchained) and development (unchained-dev) namespaces, with different timing and grouping settings to reflect operational priorities.

4. receivers

• Purpose: Defines the notification endpoints (receivers) where alerts are sent.

• Defined Receivers:

  • null — a sink that discards alerts (used as default).

  • discord_critical — sends critical alerts to a Discord webhook.

  • discord_warning — sends warning alerts to a Discord webhook.

  • discord_dev — sends both critical and warning alerts from development namespace to a Discord webhook.

• Discord Receiver Configurations:
  Each Discord receiver includes:

  • discord_configs with:

    • webhook_url: Placeholder for the actual Discord webhook URL.

    • title and message: Use Go templates (discord.title and discord.message) to format alert notifications.

5. templates

• Purpose: Specifies the file paths for Go template files used to format alert messages.

• The template files are located at /etc/alertmanager/config/*.tmpl.

Important Implementation Details

• Label Matchers:
  Alert filtering uses label match expressions such as alertname = "..." or regex matches like alertname =~ "pattern". This enables fine-grained control over alert routing.

• Grouping Logic:
  Alerts are grouped by labels to batch notifications efficiently. Grouping by resource-specific labels like statefulset, deployment, or pod prevents flooding the notification channel with individual alerts.

• Inhibition Logic:
  Inhibition prevents alert storms by suppressing lower severity alerts when a higher severity alert exists for the same resource and issue, improving signal-to-noise ratio.

• Environment Awareness:
  Alerts from production and development namespaces are routed separately with different notification cadences to align with operational needs.

• Discord Integration:
  The configuration uses Discord webhooks for alert delivery, enabling teams to receive alerts directly in designated Discord channels with rich formatting.

Interaction with Other System Components

• Prometheus:
  Prometheus evaluates alerting rules (defined externally in files like rules.json) and sends alert events to Alertmanager.

• Alertmanager:
  Uses this config.yaml to apply routing, grouping, inhibition, and notification logic.

• Discord:
  Acts as the endpoint for alert notifications, receiving formatted messages via webhook URLs.

• Alert Templates:
  Stored separately, templates define the message appearance using Go templating, enhancing readability.

• Kubernetes:
  Alerts often derive from Kubernetes resource metrics (e.g., stateful sets, pods), tying monitoring directly to cluster health.

Usage Examples

Sending a Critical Alert for StatefulSet Down in Production Namespace

When Prometheus fires an alert with labels:

alertname: UnchainedStatefulSetDown
namespace: unchained
severity: critical
statefulset: my-statefulset

• Alertmanager matches it to the route for critical alerts in unchained namespace with alertname = UnchainedStatefulSetDown.

• It groups alerts by alertname, namespace, and statefulset.

• After waiting 5 minutes (group_wait), it sends a notification via the discord_critical receiver.

• If the alert persists, repeat notifications are sent every 1 hour.

Visual Diagram

flowchart TD
    A[Prometheus] -->|Sends Alerts| B[Alertmanager]

    B -->|Applies Global Settings| C[Global Config]
    B -->|Applies Inhibition Rules| D[Inhibition Engine]
    B -->|Routes Alerts| E[Routing Tree]

    E -->|Matches Critical Alerts| F[discord_critical Receiver]
    E -->|Matches Warning Alerts| G[discord_warning Receiver]
    E -->|Matches Dev Alerts| H[discord_dev Receiver]
    E -->|Default| I[null Receiver]

    F -->|Sends Notifications| J[Discord Critical Channel]
    G -->|Sends Notifications| K[Discord Warning Channel]
    H -->|Sends Notifications| L[Discord Dev Channel]

    B -->|Formats Messages| M[Template Engine]

    classDef prod fill:#f9d5d3,stroke:#b22222,stroke-width:1px;
    classDef warn fill:#fef3c7,stroke:#b28d00,stroke-width:1px;
    classDef dev fill:#d3e5f9,stroke:#2155a4,stroke-width:1px;

    F,J prod
    G,K warn
    H,L dev

Summary

`config.yaml` is a critical configuration file for Alertmanager that:

• Defines global alert behavior.

• Implements inhibition rules to reduce duplicate notifications.

• Routes alerts to different Discord channels based on severity and environment.

• Ensures alerts are grouped efficiently.

• Uses templating for clear, structured alert messages.

• Integrates tightly with Prometheus alert rules and Kubernetes monitoring to provide actionable notifications.

This file plays a central role in the observability and incident response workflow by bridging metric-based alerting with team communication platforms.

Appendix: Key Terms

This documentation should provide a comprehensive guide for understanding and maintaining the `config.yaml` Alertmanager configuration within the ShapeShift Unchained monitoring ecosystem.