init_data.py

Overview

The init_data.py file is a utility script responsible for initializing and setting up foundational data and configurations for the InfiniFlow system's backend services. It primarily focuses on:

• Creating essential database entries such as superusers, tenants, and their roles.

• Initializing Large Language Model (LLM) factories and associated LLM configurations.

• Adding predefined graph templates used within the application.

• Synchronizing and updating knowledgebase document counts.

• Ensuring that critical default data is present to enable smooth operation of the system after deployment or reset.

This script is typically executed during the system startup or deployment process to bootstrap necessary data structures in the database.

Detailed Components

Functions

encode_to_base64(input_string: str) -> str

Encodes a given string into its Base64 representation.

• Parameters:

  • input_string (str): The string to encode.

• Returns:

  • str: The Base64 encoded string.

• Usage Example:

  encoded_password = encode_to_base64("admin")
  print(encoded_password)  # Outputs base64 string of "admin"
  

• Details:
  Uses Python's built-in base64 library to encode UTF-8 strings.

init_superuser() -> None

Initializes a default superuser account, associated tenant, tenant roles, and tenant LLM settings.

• Parameters: None

• Returns: None

• Functionality:

  • Creates a user with hardcoded credentials (email: [email protected], password: admin).

  • Creates a tenant for this user.

  • Assigns the user as the owner of the tenant.

  • Initializes default Tenant LLM configurations.

  • Verifies the functionality of critical LLM models by sending test queries.

  • Logs important info and errors during the process.

• Usage Example:

  init_superuser()
  # Creates admin user and verifies models
  

• Important Notes:

  • Password is stored encoded in Base64 (not hashed) - recommended to change after first login.

  • Integrates with services like UserService, TenantService, TenantLLMService.

  • Uses settings for default model IDs.

  • Runs a test chat and embedding encoding to verify model operability.

init_llm_factory() -> None

Initializes and cleans up the LLM factories and LLM configurations in the database.

• Parameters: None

• Returns: None

• Functionality:

  • Deletes legacy or unwanted LLM factory entries and associated LLMs (such as "MiniMax", "cohere", "Local", "Moonshot", "QAnything").

  • Iterates through configured LLM factory info (settings.FACTORY_LLM_INFOS) and inserts or updates factories and their LLMs.

  • Updates tenant LLM entries to replace deprecated factory names with current ones.

  • Updates tenant parser IDs for all tenants with a default set.

  • Adds OpenAI embedding models (text-embedding-3-small and text-embedding-3-large) to tenants that already have OpenAI LLMs.

  • Updates knowledgebase document counts for all knowledgebases.

• Usage Example:

  init_llm_factory()
  # Cleans and sets up LLM factories and models
  

• Implementation Details:

  • Uses multiple service classes for atomic CRUD operations on database models.

  • Uses a mixture of filter-delete, filter-update, and save operations.

  • Handles exceptions silently to avoid breaking the initialization flow.

add_graph_templates() -> None

Loads and inserts graph templates from a predefined directory into the database.

• Parameters: None

• Returns: None

• Functionality:

  • Deletes all existing graph templates.

  • Reads JSON files from the agent/templates directory (relative to project base).

  • Tries to save each JSON template into the database.

  • Updates existing templates if save fails.

  • Logs warnings if the templates directory is missing or if errors occur during the process.

• Usage Example:

  add_graph_templates()
  # Loads and inserts visualization templates
  

• Important Implementation Notes:

  • Utilizes CanvasTemplateService for database operations.

  • Reads templates assuming UTF-8 encoding.

  • Designed to refresh all templates on every run.

init_web_data() -> None

Main entry point to initialize web-related data components.

• Parameters: None

• Returns: None

• Functionality:

  • Calls init_llm_factory() to initialize LLM configurations.

  • Calls add_graph_templates() to load visualization templates.

  • Optionally (commented out) initializes superuser if no users exist.

  • Logs total time taken for initialization.

• Usage Example:

  init_web_data()
  # Initializes LLM factories and templates
  

File Execution Behavior

• When run as a script (main), it:

  1. Initializes web database tables by calling init_web_db().

  2. Calls init_web_data() to setup initial data.

This makes the file a convenient bootstrap utility for fresh deployments or resets.

Important Implementation Details & Algorithms

• Data Initialization Workflow:
  The file systematically cleans legacy data, inserts default and factory LLMs, sets up tenants and users, and loads necessary visualization templates, ensuring a consistent initial state.

• Model Verification:
  After creating the superuser and tenant LLMs, it sends test queries to the chat and embedding LLMs to confirm operational status, logging errors if models fail.

• Data Integrity:
  Uses service-layer abstractions (UserService, TenantService, etc.) for database operations, which encapsulate ORM logic and improve maintainability.

• Resilience:
  Most database operations are wrapped in try-except blocks to avoid interruption during initialization.

• Settings Driven:
  Relies heavily on external settings for default model IDs and factory configurations, allowing flexible environment-specific setups.

Interaction With Other System Components

• Database Models & Services:
  Interacts with database ORM models (LLMType, UserTenantRole, LLM, LLMFactories, TenantLLM) and their corresponding service classes for CRUD operations.

• Settings Module:
  Pulls default model IDs (CHAT_MDL, EMBEDDING_MDL, etc.) and factory LLM info from a central configuration.

• API Services:
  Utilizes high-level services for user, tenant, knowledgebase, and template management.

• Utility Functions:
  Uses utility methods like get_project_base_directory for path resolution.

• LLM Bundles:
  Uses LLMBundle class to interact with specific LLM instances for validation.

Visual Diagram

flowchart TD
    A[init_data.py] --> B[init_web_db()]
    A --> C[init_web_data()]
    C --> D[init_llm_factory()]
    C --> E[add_graph_templates()]
    C --> F[init_superuser()]:::optional

    D --> D1[Delete legacy LLMFactories and LLMs]
    D --> D2[Insert new LLMFactories]
    D --> D3[Insert LLMs for each factory]
    D --> D4[Update TenantLLM entries]
    D --> D5[Update Tenant parser_ids]
    D --> D6[Insert OpenAI embedding models]
    D --> D7[Update knowledgebase doc counts]

    E --> E1[Delete all CanvasTemplates]
    E --> E2[Load JSON templates from directory]
    E --> E3[Save or update templates]

    F --> F1[Create superuser entry]
    F --> F2[Create tenant entry]
    F --> F3[Create user-tenant role]
    F --> F4[Insert TenantLLMs]
    F --> F5[Test chat & embedding LLMs]

    classDef optional fill:#f9f,stroke:#333,stroke-width:2px;
    class F optional;

Summary

The init_data.py file is a crucial initialization script that prepares the InfiniFlow system's backend data environment. It ensures the presence of essential user accounts, tenants, LLM factories and models, visual templates, and knowledgebase metadata. By carefully cleaning outdated entries and inserting default configurations, it helps maintain data consistency and readiness for the system's operation. The file leverages service abstractions and settings-driven configurations to achieve a robust and flexible initialization workflow.