opensearch-config.yaml

Overview

The opensearch-config.yaml file defines a Kubernetes ConfigMap resource that configures an OpenSearch node instance within the application. This configuration is conditionally applied only when the environment variable DOC_ENGINE is set to "opensearch". The ConfigMap provides key configuration settings related to OpenSearch node identity, cluster discovery, security plugins, disk allocation thresholds, timezone, and the HTTP port.

This file is primarily used in Kubernetes deployments to inject OpenSearch-specific configuration data into pods running OpenSearch nodes, enabling the application to run OpenSearch as its search engine backend.

Detailed Explanation

Kubernetes ConfigMap

• Kind: ConfigMap

• apiVersion: v1

• Metadata Name: The name is dynamically generated using the Helm template helper ragflow.fullname appended with -opensearch-config. This ensures a unique and consistent naming convention within the Kubernetes namespace.

Data Fields

Conditional Rendering

This entire ConfigMap is rendered only if the Helm value .Values.env.DOC_ENGINE is equal to "opensearch". This allows the deployment to be flexible and support other document engines by omitting or including this ConfigMap accordingly.

Usage Example

When deploying the Helm chart with OpenSearch as the document engine, the rendered ConfigMap might look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-opensearch-config
data:
  node.name: opensearch01
  bootstrap.memory_lock: "false"
  discovery.type: single-node
  plugins.security.disabled: "false"
  plugins.security.ssl.http.enabled: "false"
  plugins.security.ssl.transport.enabled: "true"
  cluster.routing.allocation.disk.watermark.low: 5gb
  cluster.routing.allocation.disk.watermark.high: 3gb
  cluster.routing.allocation.disk.watermark.flood_stage: 2gb
  TZ: UTC
  http.port: "9201"

This ConfigMap can then be mounted as a volume or injected as environment variables into OpenSearch pods to configure the node accordingly.

Important Implementation Details

• Disk Watermarks: The cluster routing allocation disk watermarks are set with an unusual ordering (low watermark = 5GB, high watermark = 3GB, flood stage = 2GB). Normally, these thresholds increase from low to high to flood stage. This may be intentional or a misconfiguration:

  • Usual behavior:

    • low watermark < high watermark < flood stage

  • Here:

    • low watermark (5GB) > high watermark (3GB) > flood stage (2GB)

  This could potentially cause unexpected shard allocation behavior and deserves verification.

• Security Plugin Settings:

  • Security plugin is enabled (plugins.security.disabled: "false").

  • HTTP SSL is disabled (ssl.http.enabled: "false"), transport SSL enabled (ssl.transport.enabled: "true").
    This configuration means internal node-to-node communication is encrypted, but HTTP REST API is not secured by SSL.

• Single Node Discovery:
  The node runs in single-node mode, disabling multi-node cluster discovery. This is suitable for development or single-node deployments but not for production multi-node clusters.

Interaction with Other Parts of the System

• Helm Values: The file leverages Helm templating to conditionally include the configuration based on DOC_ENGINE environment variable and set the timezone.

• OpenSearch Pods: The ConfigMap is consumed by the OpenSearch pods in the Kubernetes cluster. It provides runtime configuration for OpenSearch nodes.

• Application Search Layer: If the application uses OpenSearch as a search backend, this configuration controls how OpenSearch nodes are initialized.

• Security Integration: The security plugin configurations here affect authentication and transport encryption, impacting how other services interact securely with this OpenSearch node.

Mermaid Diagram: ConfigMap Data Structure & Conditional Flow

flowchart TD
    Start[Start: Render ConfigMap?]
    Condition{Is DOC_ENGINE == "opensearch"?}
    ConfigMap[Create ConfigMap: <br> - Metadata name<br> - Data fields]
    DataNodeName[node.name: "opensearch01"]
    DataMemLock[bootstrap.memory_lock: "false"]
    DataDiscovery[discovery.type: "single-node"]
    DataSecurity[plugins.security.disabled: "false"]
    DataSSLHttp[plugins.security.ssl.http.enabled: "false"]
    DataSSLTransport[plugins.security.ssl.transport.enabled: "true"]
    DataDiskLow[cluster.routing.allocation.disk.watermark.low: 5gb]
    DataDiskHigh[cluster.routing.allocation.disk.watermark.high: 3gb]
    DataDiskFlood[cluster.routing.allocation.disk.watermark.flood_stage: 2gb]
    DataTZ[TZ: from .Values.env.TIMEZONE]
    DataHTTPPort[http.port: "9201"]
    End[End]

    Start --> Condition
    Condition -- Yes --> ConfigMap
    Condition -- No --> End
    ConfigMap --> DataNodeName
    ConfigMap --> DataMemLock
    ConfigMap --> DataDiscovery
    ConfigMap --> DataSecurity
    ConfigMap --> DataSSLHttp
    ConfigMap --> DataSSLTransport
    ConfigMap --> DataDiskLow
    ConfigMap --> DataDiskHigh
    ConfigMap --> DataDiskFlood
    ConfigMap --> DataTZ
    ConfigMap --> DataHTTPPort
    DataHTTPPort --> End

Summary

• opensearch-config.yaml is a Helm-templated Kubernetes ConfigMap for configuring an OpenSearch node.

• It activates only when DOC_ENGINE environment variable is set to "opensearch".

• Configures node name, cluster discovery, security plugins, disk watermarks, timezone, and HTTP port.

• Important to verify disk watermark values due to unusual ordering.

• Provides critical runtime configuration for OpenSearch pods in the cluster.

• Integrates with Helm values for dynamic configuration and supports secure transport between nodes.

This file is essential for managing OpenSearch node behavior in Kubernetes-based deployments of the application.