migration.sh

Overview

migration.sh is a Bash script designed to facilitate backup and restore operations for RAGFlow's Docker volumes. It manages persistent data for critical services including MySQL, MinIO, Redis, and Elasticsearch by creating compressed archive backups or restoring data from these backups into Docker volumes.

The script ensures data integrity by verifying Docker availability, checking for running containers using the target volumes, and confirming destructive operations with the user. It provides clear user guidance, error handling, and detailed status messages throughout the process.

Functionality Summary

• Backup: Archives data from specified Docker volumes into .tar.gz files inside a designated backup folder.

• Restore: Extracts archived data from backup files into Docker volumes, creating volumes if necessary.

• Safety Checks: Ensures Docker is running and no containers are actively using volumes during operations.

• User Interaction: Displays help info, prompts for confirmation on potentially destructive actions, and provides detailed logs.

• Flexible Backup Location: Allows specifying a custom backup folder, defaulting to backup/.

Detailed Explanation of Script Components

Global Variables

Functions

show_help()

Displays usage instructions, available operations, parameters, and examples.

Usage:

show_help

Output:

• Instructions on using the script.

• List of Docker volumes included.

• Examples for backing up and restoring.

check_docker()

Verifies if Docker is running and accessible. Exits with error if Docker is not available.

Usage:

check_docker

Behavior:

• Runs docker info silently.

• If Docker is not running or accessible, prints error and exits script.

volume_exists(volume_name)

Checks if a Docker volume exists.

Parameters:

• volume_name (string): Name of the Docker volume to check.

Returns:

• Exit status 0 if volume exists.

• Non-zero otherwise.

Usage Example:

if volume_exists "docker_mysql_data"; then
  echo "Volume exists"
fi

check_containers_using_volumes()

Checks if any running Docker containers are currently using any of the target volumes.

Usage:

check_containers_using_volumes

Behavior:

• Lists all running containers.

• Inspects each container's mounted volumes.

• If any container uses the target volumes, prints detailed error info and exits.

• Otherwise, confirms safe to proceed.

confirm_action(message)

Prompts the user for confirmation before proceeding.

Parameters:

• message (string): Prompt message to display.

Returns:

• Exit status 0 if user confirms (y or yes).

• Exit status 1 otherwise.

Usage Example:

if confirm_action "Do you want to continue?"; then
  echo "Proceeding..."
else
  echo "Cancelled."
fi

perform_backup(backup_folder)

Creates backup archives of the specified Docker volumes.

Parameters:

• backup_folder (string): Directory where backup archives will be stored.

Behavior:

1. Checks for any containers using target volumes.

2. Creates the backup folder if missing.

3. Iterates over all volumes:

   • Checks if volume exists.

   • Runs an Alpine Docker container to tar and compress the volume contents into a .tar.gz file in the backup folder.

4. Prints summary of created backup files with sizes.

Usage Example:

perform_backup "my_backup_folder"

perform_restore(backup_folder)

Restores data from backup archives into Docker volumes.

Parameters:

• backup_folder (string): Directory containing the backup archives.

Behavior:

1. Checks for containers using target volumes.

2. Validates the backup folder exists.

3. Checks all required backup files exist in the folder.

4. Warns if Docker volumes already exist and prompts user to confirm overwrite.

5. For each volume:

   • Creates the Docker volume if missing.

   • Restores data by extracting the archive into the volume using an Alpine container.

6. Prints completion message with recommendation to start services.

Usage Example:

perform_restore "my_backup_folder"

Main Script Logic (main() function)

Entry point of the script which:

1. Checks Docker availability.

2. Parses command-line arguments for operation (backup or restore) and optional backup folder.

3. Shows help if arguments are missing or help is requested.

4. Calls the appropriate function (perform_backup or perform_restore) based on operation.

5. Exits with error for invalid operations.

Important Implementation Details & Algorithms

• Volume Backup/Restore Using Alpine Container:
  Uses lightweight alpine Docker image to mount volumes and perform tar compression/decompression. This avoids direct host filesystem manipulation and ensures portability and isolation.

• Safety Checks Before Operations:
  The script prevents data corruption by checking that no running containers are currently using the volumes. If volumes are in use, the script aborts with instructions.

• User Confirmation on Overwrite:
  During restore, if existing volumes are detected, the user must explicitly confirm to proceed, preventing accidental data loss.

• Dynamic Backup Folder Management:
  Backup folder defaults to backup but can be overridden via command parameters.

• Error Handling & Exit on Failure:
  The script uses set -e to stop on errors but also performs manual checks to provide user-friendly error messages and guidance.

Interaction With Other System Components

• Docker Engine:
  Core dependency for volume inspection, creation, and running ephemeral containers for backup/restore.

• RAGFlow Services:
  The volumes backed up/restored are linked to RAGFlow's key components: MySQL, MinIO, Redis, Elasticsearch. These services rely on persistent data stored in these volumes.

• Docker Compose:
  The script suggests using docker-compose commands to stop/start containers before and after migration, indicating integration with docker-compose-managed environments.

• Filesystem:
  Backup archives are stored on the host filesystem, enabling offsite storage or transfer.

Usage Examples

• Backup using default folder:

  ./migration.sh backup
  

• Backup using custom folder:

  ./migration.sh backup my_backup_2024
  

• Restore using default folder:

  ./migration.sh restore
  

• Restore using custom folder:

  ./migration.sh restore my_backup_2024
  

• Show help:

  ./migration.sh help
  

Mermaid Flowchart Diagram: Workflow of migration.sh

flowchart TD
    A[Start Script] --> B{Parse Operation}
    B -->|help| C[show_help()]
    B -->|backup| D[perform_backup()]
    B -->|restore| E[perform_restore()]
    B -->|invalid| F[Show Error + show_help()]

    D --> D1[check_docker()]
    D --> D2[check_containers_using_volumes()]
    D --> D3[Create backup folder if missing]
    D --> D4[For each volume]
    D4 --> D4a[Check if volume exists]
    D4a --> D4b[Run alpine container to tar volume]
    D4b --> D4c[Save archive to backup folder]
    D4c --> D5[Print backup summary]

    E --> E1[check_docker()]
    E --> E2[check_containers_using_volumes()]
    E --> E3[Validate backup folder & files]
    E --> E4[Check existing volumes]
    E4 -->|exists| E5[Prompt user to confirm overwrite]
    E --> E6[For each volume]
    E6 --> E6a[Create volume if missing]
    E6a --> E6b[Run alpine container to extract archive]
    E6b --> E7[Print restore summary]

    C --> G[Exit]
    F --> G
    D5 --> G
    E7 --> G

Summary

The migration.sh script is a robust and user-friendly utility for managing RAGFlow persistent data by backing up and restoring Docker volumes safely. It integrates tightly with Docker, enforces safety checks, and guides users through backup and restore workflows with clear instructions and feedback. The modular design with discrete functions enhances maintainability and clarity.

End of documentation for migration.sh