log_utils.py

Overview

log_utils.py is a utility module designed to facilitate consistent and robust logging across the InfiniFlow project. It provides functions to initialize the root logger with configurable log file handling, console output, and dynamic log level settings. The module ensures that logs are rotated to prevent excessive file sizes and supports environment-driven log level customization for different packages. Additionally, it offers a helper function to log exceptions along with contextual messages.

By encapsulating logging setup and exception logging logic, this module promotes standardized logging practices, aiding in debugging, monitoring, and maintenance of the application.

Detailed Documentation

Global Variables

• initialized_root_logger (bool):
  Tracks whether the root logger has been initialized to avoid duplicate setups.

Functions

get_project_base_directory() -> str

Returns the absolute path to the base directory of the project by traversing two levels up from the current file's location.

• Returns:
  str — Absolute path of the project base directory.

• Usage Example:

  base_dir = get_project_base_directory()
  print("Project base directory:", base_dir)
  

• Implementation Details:
  Uses os.path.realpath(file) to get the current file location, then moves up two directories using os.pardir to reach the root of the project.

init_root_logger(logfile_basename: str, log_format: str = "%(asctime)-15s %(levelname)-8s %(process)d %(message)s") -> None

Initializes the root logger with two handlers: a rotating file handler and a console stream handler. Supports dynamic log levels per package via the environment variable LOG_LEVELS.

• Parameters:

  • logfile_basename (str): Base name for the log file. The log file will be created at <project_base>/logs/{logfile_basename}.log.

  • log_format (str, optional): Format string for log messages. Defaults to a format including timestamp, log level, process ID, and message.

• Returns:
  None

• Usage Example:

  from log_utils import init_root_logger
  
  init_root_logger("infini_app")
  logging.info("Logger initialized successfully.")
  

• Implementation Details:

  • Ensures only one initialization (singleton pattern using initialized_root_logger flag).

  • Clears any existing handlers on the root logger before adding new ones.

  • Creates a logs directory inside the project base directory if it does not exist.

  • Sets up a RotatingFileHandler with a 10MB max file size and 5 backup files to handle log rotation.

  • Adds a StreamHandler to output logs to the console.

  • Calls logging.captureWarnings(True) to redirect Python warnings to the logging system.

  • Parses LOG_LEVELS environment variable to set log levels for individual packages in the format "package=LEVEL,package2=LEVEL2".

  • Defaults peewee and pdfminer packages to WARNING level if not specified.

  • Defaults root logger level to INFO if not set.

  • Logs the path of the logfile and the configured log levels upon setup.

log_exception(e: Exception, *args) -> None

Logs an exception with traceback and additional context messages or objects, then re-raises the original exception.

• Parameters:

  • e (Exception): The exception instance to log.

  • *args (variable length): Additional context arguments. If an argument has a text attribute, logs its content and raises a new Exception with that text. Otherwise, logs the string representation of the argument.

• Returns:
  None (raises the original or new exception after logging).

• Usage Example:

  try:
      # some code that might raise
      risky_operation()
  except Exception as ex:
      log_exception(ex, "Additional context message")
  

• Implementation Details:

  • Uses logging.exception() to log the stack trace of the exception.

  • Iterates over additional arguments to log extra context.

  • If an argument has a .text attribute (e.g., a response object), logs the text and raises a new Exception with that message.

  • Otherwise, logs the stringified argument.

  • Finally, re-raises the original exception e.

Implementation and Algorithm Notes

• Log Rotation:
  Utilizes Python’s RotatingFileHandler to avoid unbounded log file growth, limiting each log file to 10MB and keeping 5 backup files.

• Dynamic Log Levels:
  Environment variable LOG_LEVELS allows runtime configuration of logging verbosity per package without code changes. This feature supports flexible debugging and monitoring.

• Singleton Logger Initialization:
  Prevents multiple logger setups by using a module-level flag initialized_root_logger.

• Exception Context Handling:
  log_exception enhances error reporting by allowing additional context objects or messages to be logged, improving traceability.

Interaction with Other Parts of the System

• Project Base Directory Usage:
  get_project_base_directory is used internally to locate the root of the project, ensuring log files are stored in a consistent location relative to the project.

• Logging Setup:
  The initialized root logger affects all modules since Python’s logging is hierarchical. Other modules in InfiniFlow can rely on this centralized setup for consistent log formatting and level control.

• Package Log Levels:
  The module explicitly sets log levels for critical dependencies like peewee (an ORM) and pdfminer (PDF processing), impacting how verbose logs from those libraries appear in the system.

• Exception Logging:
  log_exception is meant to be called from exception handlers throughout the application to standardize error logging and ensure relevant context is recorded.

Visual Diagram

flowchart TD
    A[get_project_base_directory]
    B[init_root_logger]
    C[log_exception]

    B -->|calls| A
    B -->|sets up| D[Root Logger]
    B -->|adds| E[RotatingFileHandler]
    B -->|adds| F[StreamHandler]
    B -->|reads| G[Environment Variable LOG_LEVELS]
    B -->|sets levels for| H[Package Loggers (peewee, pdfminer, root)]
    C -->|logs| D

Summary

The log_utils.py module is a foundational utility for managing logging in the InfiniFlow project. It provides a streamlined way to initialize the root logger with file rotation, console output, and environment-driven log levels, while also offering a robust exception logging function that captures and enriches error information. This utility ensures that logs are both manageable and informative, facilitating easier debugging and operational insight.