searxng.py

Overview

The searxng.py file defines a component that interfaces with SearXNG, a privacy-focused metasearch engine which aggregates search results from multiple sources without tracking users. This component enables querying a SearXNG instance programmatically, processing and limiting the results, and returning them in a structured format for further use within the application.

This file includes parameter definitions and the main logic to invoke SearXNG’s search API, handling network errors gracefully and supporting configurable options such as the number of results to return and the SearXNG server URL.

Classes and Their Functionalities

1. SearXNGParam(ToolParamBase)

Defines configurable parameters used by the SearXNG component.

Properties:

• meta: ToolMeta
  Metadata describing the tool including:

  • name: Identifier of the tool ("searxng_search").

  • description: Overview of SearXNG and its purpose.

  • parameters: Dictionary defining input parameters:

    • query: Search keywords (string, required).

    • searxng_url: Base URL of the SearXNG instance (string, optional).

• top_n: int
  Maximum number of search results to return (default: 10).

• searxng_url: str
  Stores the SearXNG server URL (default empty).

Methods:

• init(self)
  Initializes the parameter object with default values.

• check(self)
  Validates parameters:

  • Attempts to coerce top_n to an integer if given as a string.

  • Ensures top_n is a positive integer.

  • Validation is lenient to allow the component to be instantiated without a URL (for try-run scenarios).

• get_input_form(self) -> dict[str, dict]
  Returns a dictionary defining the input form for UI or API interaction:

{
    "query": {"name": "Query", "type": "line"},
    "searxng_url": {"name": "SearXNG URL", "type": "line", "placeholder": "http://localhost:4000"}
}

2. SearXNG(ToolBase, ABC)

Main component class that performs the actual search on a SearXNG instance.

Properties:

• component_name: str
  Static name identifying the component: "SearXNG".

Methods:

• _invoke(self, **kwargs)
  The core method executing a search query against SearXNG.

  Parameters:

  • kwargs: Expected keys include:

    • "query" (str): The search query string.

    • "searxng_url" (str, optional): URL of the SearXNG instance.

  Returns:

  • The formalized content output as a string (aggregated search results).

  • Empty string if inputs are invalid or URL is missing during try-run.

  • Error message string on repeated failure.

  Behavior:

  1. Validates the presence and format of the search query.

  2. Determines the SearXNG server URL from input, parameters, or defaults.

  3. Attempts to perform HTTP GET requests to /search endpoint of the SearXNG instance with parameters:

     • q: the query string.

     • format: "json"

     • categories: "general"

     • language: "auto"

     • safesearch: 1 (enabled)

     • pageno: 1

  4. Parses and validates the JSON response.

  5. Extracts and limits the results to top_n.

  6. Processes each result using _retrieve_chunks (inherited method) extracting title, URL, and content.

  7. Sets outputs for downstream consumption and returns the processed content.

  8. Retries on network or processing errors up to max_retries + 1, with delays between attempts.

  9. Logs exceptions and sets error output on failure.

  Example Usage:

searxng_component = SearXNG()
outputs = searxng_component._invoke(query="privacy focused search engines", searxng_url="http://localhost:4000")
print(outputs)

• thoughts(self) -> str
  Returns a string describing the current operation or internal state, used for logging or debugging.

  Example Output:

Keywords: privacy focused search engines 
Searching with SearXNG for relevant results...

Important Implementation Details

• Timeout Handling:
  The _invoke method is decorated with @timeout which enforces a maximum execution time (default 12 seconds, configurable via environment variable COMPONENT_EXEC_TIMEOUT).

• Error Handling & Retry Logic:
  The component retries failed searches due to network or unexpected errors, implementing a delay after each error (self._param.delay_after_error). This increases robustness in unstable network environments.

• Result Processing:
  Results are sliced to the top N (default 10), ensuring the system does not overload downstream processing with excessive data.

• Flexible Parameter Sourcing:
  The SearXNG URL can come from multiple sources: method arguments, stored parameters, or defaults. This flexibility allows integration in different deployment contexts.

• Internationalization & Search Safety:
  Search requests specify language auto-detection and safesearch enabled to filter inappropriate content.

Interaction with Other System Components

• Inheritance:

  • SearXNGParam extends ToolParamBase from agent.tools.base which likely provides basic parameter validation and management utilities.

  • SearXNG extends ToolBase and ABC (Abstract Base Class), inheriting common tool behavior and interfaces for the InfiniFlow agent framework.

• Utility Decorators:
  Uses the timeout decorator from api.utils.api_utils to manage execution time limits.

• HTTP Requests:
  Uses the requests library for HTTP communication with the SearXNG API.

• Logging:
  Integrates Python's logging module to record errors and tracebacks for debugging.

• Output Handling:
  Uses methods like set_output and output (from ToolBase) to manage data passed to other components or the user interface.

Visual Diagram: Class Structure

classDiagram
    class SearXNGParam {
        +meta: ToolMeta
        +top_n: int
        +searxng_url: str
        +__init__()
        +check()
        +get_input_form() dict
    }

    class SearXNG {
        +component_name: str
        +_invoke(kwargs) str
        +thoughts() str
    }

    class ToolParamBase
    class ToolBase
    class ABC

    SearXNGParam --|> ToolParamBase
    SearXNG --|> ToolBase
    SearXNG --|> ABC

Summary

searxng.py is a well-structured, fault-tolerant component for integrating SearXNG search capabilities into a larger system. It abstracts complexity around parameter handling, API communication, error management, and result processing, making it straightforward to extend or embed SearXNG metasearch functionality in diverse applications within the InfiniFlow framework.