list_ignore_probes.sh

Overview

`list_ignore_probes.sh` is a Bash utility script designed to scan Kubernetes pods within specific namespaces, searching for certain "ignore probe" indicator files named with the pattern `disable_*`. These files, if present, likely serve as flags to disable or ignore health probes or related checks within containerized applications.

The script automates the process of:

• Enumerating StatefulSets in predefined namespaces.

• Retrieving pods associated with these StatefulSets via label selectors.

• Iterating through all containers in each pod.

• Executing commands inside containers to find files matching the disable_* pattern within /data and /root directories.

• Printing out the discovered files along with their namespace, pod, and container context.

This tool is useful for operations or DevOps teams who need to audit or debug probe disabling flags across a Kubernetes cluster to ensure that ignore probes are accounted for or to diagnose unexpected probe behavior.

Detailed Explanation

Script Structure and Workflow

The script operates in a nested loop structure:

1. Namespace Iteration: Loops over a fixed list of namespaces (unchained-dev and unchained).

2. StatefulSet Retrieval: For each namespace, fetches the names of all StatefulSets.

3. Pods Retrieval: For each StatefulSet, it extracts label selectors and uses them to find all pods matching those labels.

4. Container Enumeration: For each pod, lists all containers.

5. File Search: Inside each container, runs find commands to locate files named disable_* under /data and /root.

6. Output: Prints any discovered files with contextual information.

Variables

• namespaces: An array containing the target Kubernetes namespaces to scan.

Commands and Key Operations

• kubectl get statefulsets -n $ns -o jsonpath='{.items[*].metadata.name}'
  Retrieves StatefulSet names in the namespace.

• kubectl get statefulset $ss -n $ns -o jsonpath='{.spec.selector.matchLabels}'
  Extracts the label selectors from the StatefulSet spec.

• jq -r 'to_entries | map("\(.key)=\(.value|tostring)") | join(",")'
  Converts the JSON label selector object into a comma-separated label selector string for kubectl.

• kubectl get pods -n $ns -l $label_selector -o jsonpath='{.items[*].metadata.name}'
  Lists pods matching the label selector in the namespace.

• kubectl get pod $pod -n $ns -o jsonpath='{.spec.containers[*].name}'
  Retrieves the container names within the pod.

• kubectl exec -it $pod -n $ns -c $container -- find /data -type f -name "disable_*"
  Executes the find command inside the container to look for files named with disable_* pattern.

Important Implementation Details

• Label Selector Construction:
  The script uses the StatefulSet's .spec.selector.matchLabels to dynamically build the label selector string, ensuring accurate pod discovery associated with that StatefulSet.

• Error Handling:
  The script filters out cases where the find command returns "No such file or directory" errors by checking the output before printing results.

• Use of kubectl exec:
  Executes commands inside containers with -it flags to enable interactive and TTY modes, which may be unnecessary and could be adjusted for scripting robustness.

• Search Locations:
  Searches are limited to /data and /root directories, likely because these are known locations where probe disabling files may reside.

Functions / Methods

This script does not define explicit functions or classes; it is a linear procedural script using loops and command invocations.

Usage Example

Run the script in an environment with `kubectl` configured and access to the Kubernetes cluster:

./list_ignore_probes.sh

Expected output:

checking unchained-dev/pod-name/container-name
found unchained-dev/pod-name/container-name: /data/disable_liveness_probe
checking unchained-dev/pod-name/container-name2
checking unchained/pod-name2/container-name
...

This output shows which containers have the `disable_*` files and their paths.

Interaction with Other System Components

• Kubernetes Cluster:
  Relies on kubectl CLI access to the cluster to retrieve StatefulSets, pods, and to execute commands inside containers.

• JSON Processing:
  Uses jq to parse JSON output from kubectl and format label selectors.

• Container Filesystems:
  Inspects containers' internal filesystems for the presence of probe-disabling files.

This script is likely part of a larger operational toolkit for cluster inspection, debugging, or compliance validation related to health probes or monitoring configurations.

Mermaid Flowchart Diagram

flowchart TD
    Start --> NamespaceLoop
    NamespaceLoop["For each namespace in (unchained-dev, unchained)"] --> StatefulSetGet
    StatefulSetGet["Get StatefulSets in namespace"] --> StatefulSetLoop
    StatefulSetLoop["For each StatefulSet"] --> LabelSelectorGet
    LabelSelectorGet["Get label selector from StatefulSet"] --> PodGet
    PodGet["Get pods matching label selector"] --> PodLoop
    PodLoop["For each pod"] --> ContainerGet
    ContainerGet["Get containers in pod"] --> ContainerLoop
    ContainerLoop["For each container"] --> FindInData
    FindInData["Find disable_* files in /data"] --> CheckDataFiles
    CheckDataFiles{"Files found?"}
    CheckDataFiles -- Yes --> PrintDataFiles
    CheckDataFiles -- No --> FindInRoot
    PrintDataFiles --> FindInRoot
    FindInRoot["Find disable_* files in /root"] --> CheckRootFiles
    CheckRootFiles{"Files found?"}
    CheckRootFiles -- Yes --> PrintRootFiles
    CheckRootFiles -- No --> ContainerLoopEnd
    PrintRootFiles --> ContainerLoopEnd
    ContainerLoopEnd --> ContainerLoop
    ContainerLoop -- All containers done --> PodLoop
    PodLoop -- All pods done --> StatefulSetLoop
    StatefulSetLoop -- All StatefulSets done --> NamespaceLoop
    NamespaceLoop -- All namespaces done --> End
    End["End"]

Summary

`list_ignore_probes.sh` is a specialized Bash script for Kubernetes cluster inspection, focusing on identifying files that disable health or readiness probes inside containers of StatefulSets within specified namespaces. It leverages Kubernetes API querying via `kubectl`, JSON parsing with `jq`, and container exec commands to locate these files, providing useful insights for cluster maintenance and debugging.