common.py

Overview

The common.py file serves as a centralized client-side API utility module for interacting with multiple backend services of the InfiniFlow system. It provides a collection of Python functions that encapsulate HTTP requests targeting various RESTful endpoints related to knowledge bases (KB), documents, chunks, and dialogs.

Each group of functions corresponds to one of the core applications or services within the system:

• KB App: Manage knowledge bases and their metadata, including tags and knowledge graphs.

• Document App: Handle uploading, creating, listing, parsing, and deleting documents.

• Chunk App: Manage text chunks extracted from documents, including CRUD operations and retrieval testing.

• Dialog App: Manage dialog sessions including creation, update, retrieval, listing, and deletion.

This modular approach abstracts the low-level API calls, streamlining integration and testing for clients by providing reusable functions with consistent request patterns and error handling (via response JSON parsing).

Detailed Descriptions

Constants and Imports

• HEADERS: Default HTTP header specifying JSON content type.

• URL constants (KB_APP_URL, DOCUMENT_APP_URL, etc.): Constructed from a base VERSION string imported from configs, defining API endpoint prefixes.

• Dependencies:

  • requests: For HTTP calls.

  • MultipartEncoder from requests_toolbelt: For multipart form data handling during file uploads.

  • create_txt_file from utils.file_utils: Helper for generating temporary text files (used in bulk uploads).

KB APP Functions

These functions interact with the Knowledge Base (KB) backend service.

create_kb(auth, payload=None, *, headers=HEADERS, data=None)

• Purpose: Create a new knowledge base.

• Parameters:

  • auth: Authentication credentials (e.g., tuple or requests auth object).

  • payload: Dictionary with KB creation parameters (e.g., name).

  • headers: Optional HTTP headers.

  • data: Optional raw data.

• Returns: JSON response from the server.

• Usage:

  auth = ('user', 'pass')
  response = create_kb(auth, {"name": "new_kb"})
  print(response["data"]["kb_id"])
  

list_kbs(auth, params=None, payload=None, *, headers=HEADERS, data=None)

• Purpose: List all knowledge bases, optionally filtered.

• Parameters: Similar to create_kb.

• Returns: JSON listing KBs.

• Note: Defaults payload to empty dict if None.

update_kb(auth, payload=None, *, headers=HEADERS, data=None)

• Purpose: Update an existing KB.

• Parameters: Same as create_kb.

• Returns: JSON response.

rm_kb(auth, payload=None, *, headers=HEADERS, data=None)

• Purpose: Remove a KB.

• Parameters: Same as create_kb.

• Returns: JSON response.

detail_kb(auth, params=None, *, headers=HEADERS)

• Purpose: Retrieve details of a specific KB.

• Parameters:

  • params: Query parameters to specify KB id or filters.

• Returns: JSON with KB details.

Tag and Knowledge Graph Management Functions

• list_tags_from_kbs(auth, params=None, *, headers=HEADERS): Lists tags across KBs.

• list_tags(auth, dataset_id, params=None, *, headers=HEADERS): Lists tags for a specific dataset.

• rm_tags(auth, dataset_id, payload=None, *, headers=HEADERS, data=None): Remove tags from a dataset.

• rename_tags(auth, dataset_id, payload=None, *, headers=HEADERS, data=None): Rename tags within a dataset.

• knowledge_graph(auth, dataset_id, params=None, *, headers=HEADERS): Retrieve knowledge graph data.

• delete_knowledge_graph(auth, dataset_id, payload=None, *, headers=HEADERS, data=None): Delete knowledge graph.

batch_create_datasets(auth, num)

• Purpose: Utility to create multiple KBs rapidly, naming them sequentially (kb_0, kb_1, ...).

• Parameters:

  • auth: Auth credentials.

  • num: Number of KBs to create.

• Returns: List of created KB IDs.

• Implementation Detail: Uses create_kb in a loop.

DOCUMENT APP Functions

upload_documents(auth, payload=None, files_path=None)

• Purpose: Upload one or more document files to the server.

• Parameters:

  • auth: Auth credentials.

  • payload: Metadata or parameters for upload.

  • files_path: List of file paths to upload.

• Returns: JSON response with upload results.

• Implementation Details:

  • Uses MultipartEncoder to build multipart/form-data requests.

  • Opens files in binary mode and ensures proper closure in finally block.

• Usage:

  response = upload_documents(auth, {"kb_id": 123}, ["doc1.txt", "doc2.txt"])
  print(response)
  

Other document-related functions:

• create_document: Create a document record.

• list_documents: List documents with optional filters.

• delete_document: Delete specified documents.

• parse_documents: Trigger parsing of documents.

• bulk_upload_documents(auth, kb_id, num, tmp_path): Create multiple dummy text files and upload them in bulk.

CHUNK APP Functions

These are for managing document chunks (text fragments).

• add_chunk: Create a new chunk.

• list_chunks: List chunks.

• get_chunk: Retrieve specific chunk details.

• update_chunk: Update chunk content or metadata.

• delete_chunks: Remove chunks.

• retrieval_chunks: Perform retrieval tests on chunks.

• batch_add_chunks(auth, doc_id, num): Add multiple chunks to a document in a loop.

DIALOG APP Functions

Functions related to dialog session management.

• create_dialog: Create a new dialog session.

• update_dialog: Update an existing dialog.

• get_dialog: Get dialog details.

• list_dialogs: List all dialogs.

• delete_dialog: Delete dialogs.

• batch_create_dialogs(auth, num, kb_ids=None): Batch create dialogs with default configuration and optionally linked KB IDs.

• delete_dialogs(auth): Delete all dialogs retrieved from listing.

Important Implementation Details

• The module relies heavily on HTTP POST and GET requests with JSON payloads and responses.

• Authentication is passed as a parameter (auth) and delegated to the requests library, supporting multiple auth methods.

• The use of requests_toolbelt.MultipartEncoder allows file upload with multipart/form-data encoding.

• Batch functions automate repetitive creation steps, useful for testing or initializing datasets.

• All functions return the JSON-decoded response, expecting the server to reply with JSON.

Interaction with Other System Components

• Configurations: Reads HOST_ADDRESS and VERSION from a configs module to construct endpoint URLs dynamically.

• File Utilities: Uses create_txt_file from utils.file_utils to generate temporary files for bulk upload tests.

• Backend Services: Acts as a thin client SDK for various backend microservices managing knowledge bases, documents, chunks, and dialogs.

• This utility file is intended for use by higher-level application logic, CLI tools, or automated scripts that need to interface with the InfiniFlow backend.

Visual Diagram

The following flowchart illustrates the main functions grouped by application area and their relationships (e.g., batch functions call individual create/add functions):

flowchart TD
    subgraph KB_App [KB App]
        create_kb
        list_kbs
        update_kb
        rm_kb
        detail_kb
        list_tags_from_kbs
        list_tags
        rm_tags
        rename_tags
        knowledge_graph
        delete_knowledge_graph
        batch_create_datasets --> create_kb
    end

    subgraph Document_App [Document App]
        upload_documents
        create_document
        list_documents
        delete_document
        parse_documents
        bulk_upload_documents --> upload_documents
    end

    subgraph Chunk_App [Chunk App]
        add_chunk
        list_chunks
        get_chunk
        update_chunk
        delete_chunks
        retrieval_chunks
        batch_add_chunks --> add_chunk
    end

    subgraph Dialog_App [Dialog App]
        create_dialog
        update_dialog
        get_dialog
        list_dialogs
        delete_dialog
        batch_create_dialogs --> create_dialog
        delete_dialogs --> list_dialogs & delete_dialog
    end

Summary

common.py is a comprehensive API client layer that abstracts RESTful interactions with InfiniFlow backend services related to knowledge bases, documents, chunks, and dialogs. It provides synchronous, reusable Python functions encapsulating request logic, authentication, and response handling, thereby enabling efficient integration and automation workflows within the InfiniFlow ecosystem.