opendal_conn.py

Overview

The opendal_conn.py module provides an abstraction layer for interacting with OpenDAL (Open Data Abstraction Layer) storage systems, with a specialized focus on MySQL-based backend storage. It is designed to initialize, configure, and manage connections to OpenDAL-compatible storage, including setup of MySQL database tables and configuration parameters necessary for efficient storage handling.

This module encapsulates the complexities of configuring OpenDAL connections, especially when using MySQL as the underlying storage scheme, and exposes a singleton class OpenDALStorage that supports common storage operations like reading, writing, deleting, scanning, and checking existence of objects in the storage.

Detailed Documentation

Constants

• CREATE_TABLE_SQL
  SQL statement template for creating a storage table in MySQL if it does not already exist.
  The table has the following schema:

  • key (VARCHAR(255)): Primary key, representing the object key.

  • value (LONGBLOB): Binary large object storing the data.

  • created_at (TIMESTAMP): Timestamp of row creation, defaults to current time.

  • updated_at (TIMESTAMP): Timestamp updated on every row modification.

• SET_MAX_ALLOWED_PACKET_SQL
  SQL command template for setting the MySQL global max_allowed_packet size to allow large packets for storage.

Functions

get_opendal_config() -> dict

Fetches and prepares the configuration dictionary required to initialize an OpenDAL Operator. The configuration is read from a base configuration source (via get_base_config), supporting dynamic detection of the storage scheme.

• For the MySQL scheme:

  • Retrieves MySQL connection parameters including host, port, user, password, database name, and max allowed packet size.

  • Constructs a MySQL connection string with URL-encoded password.

  • Includes the table name used for storage.

• For other schemes:

  • Directly uses configuration keys provided.

Returns:
A dictionary containing all necessary parameters to instantiate an OpenDAL Operator.

Raises:
Any exceptions encountered during configuration loading are logged and re-raised.

Example Usage:

config = get_opendal_config()
operator = opendal.Operator(**config)

Classes

OpenDALStorage

A singleton class encapsulating the OpenDAL operator instance and providing a high-level API for common storage operations.

Singleton Decorator:
Ensures a single instance of OpenDALStorage exists throughout the application lifecycle.

Initialization

OpenDALStorage()

• Loads configuration via get_opendal_config().

• If the scheme is MySQL, performs:

  • Initialization of MySQL database configuration (init_db_config).

  • Ensures the MySQL storage table exists (init_opendal_mysql_table).

• Creates an OpenDAL Operator instance with the loaded configuration.

Methods

Method Details and Usage Examples

• health()

  Checks storage availability by writing a small test object.

  storage = OpenDALStorage()
  if storage.health():
      print("Storage is healthy")
  else:
      print("Storage health check failed")
  

• put(bucket, fnm, binary)

  Puts binary data into the storage under a given bucket and filename.

  storage.put('mybucket', 'file1.txt', b'Hello, OpenDAL!')
  

• get(bucket, fnm)

  Retrieves binary data from storage.

  content = storage.get('mybucket', 'file1.txt')
  print(content.decode('utf-8'))
  

• rm(bucket, fnm)

  Deletes the specified object and refreshes the operator connection.

  storage.rm('mybucket', 'file1.txt')
  

• scan(bucket, fnm)

  Iterates over stored objects matching the prefix.

  for key in storage.scan('mybucket', ''):
      print(key)
  

• obj_exist(bucket, fnm)

  Checks existence of an object.

  exists = storage.obj_exist('mybucket', 'file1.txt')
  print(f"Object exists: {exists}")
  

• init_db_config()

  Connects to MySQL and sets the max_allowed_packet parameter to allow large data transmission.

  Raises exceptions on failure.

• init_opendal_mysql_table()

  Ensures the MySQL table used by OpenDAL exists, creating it if necessary.

Implementation Details

• Uses pymysql for direct MySQL connection and setup.

• Uses the opendal library to instantiate an operator that abstracts storage access.

• Dynamically sets MySQL max_allowed_packet parameter to support large binary objects.

• Uses singleton pattern (@singleton decorator from rag.utils) to avoid multiple instantiations.

• Employs URL encoding (quote_plus) to safely embed passwords in connection strings.

• Logs all major steps and errors using the standard logging module.

• Supports multiple storage schemes, but has special handling for MySQL.

Integration and Usage in the System

• Relies on external get_base_config utility (from api.utils) to fetch configuration from YAML or environment.

• Exposes a singleton OpenDALStorage instance to be imported and used wherever OpenDAL storage interaction is required.

• Can be used as a backend storage abstraction for applications needing persistent binary object storage with a unified API.

• Initializes its storage table and config as part of the application startup or first usage.

Visual Diagram

classDiagram
    class OpenDALStorage {
        -_kwargs: dict
        -_scheme: str
        -_operator: opendal.Operator
        +__init__()
        +health(): bool
        +put(bucket: str, fnm: str, binary: bytes)
        +get(bucket: str, fnm: str) : bytes
        +rm(bucket: str, fnm: str)
        +scan(bucket: str, fnm: str) : Iterator
        +obj_exist(bucket: str, fnm: str) : bool
        -init_db_config()
        -init_opendal_mysql_table()
    }

    OpenDALStorage ..> opendal.Operator : uses
    OpenDALStorage ..> pymysql.Connection : uses (for DB config)
    OpenDALStorage ..> get_opendal_config : calls

    class get_opendal_config {
        +() : dict
    }

Summary

The opendal_conn.py module provides a robust and configurable interface to OpenDAL storage systems, with special support for MySQL backend configuration and table initialization. It simplifies usage of OpenDAL through a singleton class, abstracts connection details, and ensures the environment is properly configured for handling large binary objects. This module is essential in systems requiring reliable, scalable object storage with a consistent API.