invoke.py

Overview

The invoke.py file defines a reusable component within the InfiniFlow system that enables making HTTP requests to external endpoints. It provides a flexible interface for invoking web services using various HTTP methods (GET, POST, PUT), supports proxy configurations, custom headers, and flexible data formats (json or formdata). Additionally, it can optionally clean and parse HTML responses to extract meaningful content.

This component is designed to interact with remote APIs or web services as part of a larger workflow, allowing dynamic variable substitution and robust error handling with retry mechanisms.

Classes and Their Functionality

InvokeParam

Purpose:
Defines and validates configuration parameters for the Invoke component.

Inheritance:
Inherits from ComponentParamBase.

Attributes:

Methods:

• check(self):
  Validates parameters for correctness, including:

  • method is one of ['get', 'post', 'put']

  • url is not empty

  • timeout is a positive integer

  • clean_html is a boolean

  • datatype is one of ['json', 'formdata']

Usage Example:

param = InvokeParam()
param.url = "https://api.example.com/data"
param.method = "post"
param.headers = '{"Authorization": "Bearer token"}'
param.variables = [{"key": "param1", "value": "value1"}]
param.timeout = 30
param.clean_html = False
param.datatype = "json"
param.check()

Invoke

Purpose:
Main component class responsible for executing HTTP requests based on parameters and returning processed results.

Inheritance:
Inherits from ComponentBase and ABC (Abstract Base Class).

Class Attribute:

• component_name = "Invoke"

Key Method:

_invoke(self, **kwargs)

• Description:
  Core method that performs the HTTP request using the configured parameters. Supports dynamic variable injection, multiple retries, optional HTML cleaning, and handles proxies and headers.

• Decorators:
  @timeout(os.environ.get("COMPONENT_EXEC_TIMEOUT", 3))
  Enforces an execution timeout determined by an environment variable (default 3 seconds).

• Parameters:
  Accepts arbitrary keyword arguments (not used internally but allows flexibility).

• Workflow:

  1. Prepare Request Parameters:
     Iterates over self._param.variables to build a dictionary of key-value pairs for the request. Values can be static or reference dynamic canvas variables.

  2. Normalize URL:
     Ensures the URL starts with http:// if no scheme is specified.

  3. Parse Headers:
     Converts JSON string headers into a dictionary.

  4. Setup Proxy:
     If a proxy string is provided, prepares it for the requests library.

  5. Retry Loop:
     Attempts the HTTP request up to max_retries + 1 times on failure, with delays between retries.

  6. Make HTTP Request:
     Depending on method and datatype, uses appropriate requests method (get, post, put) with JSON or form data.

  7. Process Response:

     • If clean_html is True, parses the HTML content via HtmlParser and returns cleaned sections as the result.

     • Otherwise, returns the raw response text.

  8. Error Handling:
     Logs exceptions and ultimately sets an error output if all retries fail.

• Return Value:
  Returns the processed response content as a string, or error message on failure.

• Example Usage:

invoke_component = Invoke()
invoke_component._param = InvokeParam()
invoke_component._param.url = "https://jsonplaceholder.typicode.com/posts"
invoke_component._param.method = "get"
invoke_component._param.variables = [{"key": "userId", "value": "1"}]
invoke_component._param.headers = '{"Accept": "application/json"}'
invoke_component._param.timeout = 10

result = invoke_component._invoke()
print(result)  # JSON response from the API

thoughts(self) -> str

• Description:
  Returns a simple status message indicating that the component is awaiting server response.

• Return:
  "Waiting for the server respond..."

Important Implementation Details

• Dynamic Variable Resolution:
  Request parameters can be specified either as fixed values or references to external variables managed by the _canvas object, allowing integration into workflows with dynamic inputs.

• Flexible Data Posting:
  Supports both JSON and form-data POST/PUT requests, controlled by the datatype parameter.

• HTML Response Cleaning:
  When clean_html is enabled, the raw HTML response is parsed and split into meaningful sections using HtmlParser (imported from deepdoc.parser), which can be useful for extracting content from web pages.

• Proxy Support:
  Proxy URLs can be specified and are applied for both HTTP and HTTPS requests.

• Timeout and Retries:
  The _invoke method is guarded by a configurable timeout decorator and attempts retries with delays on HTTP request failures to enhance reliability.

• Error Reporting:
  Captured exceptions are logged and set as output under the key _ERROR to assist in troubleshooting.

Interaction with Other System Components

• ComponentBase and ComponentParamBase:
  Invoke and InvokeParam extend base classes that provide common component functionality such as parameter validation, output setting, and variable management.

• _canvas:
  The Invoke component accesses a _canvas object to resolve variable references dynamically, enabling integration into larger workflows or pipelines.

• HtmlParser (from deepdoc.parser):
  Used to parse and clean HTML content when requested.

• api.utils.api_utils.timeout:
  Provides the timeout decorator enforcing execution time limits.

• requests Library:
  Handles all HTTP communication.

Visual Diagram: Class Structure of invoke.py

classDiagram
    class InvokeParam {
        +proxy: str or None
        +headers: str
        +method: str
        +variables: list
        +url: str
        +timeout: int
        +clean_html: bool
        +datatype: str
        +check()
    }

    class Invoke {
        +component_name: str
        +_invoke(**kwargs) str
        +thoughts() str
    }

    ComponentParamBase <|-- InvokeParam
    ComponentBase <|-- Invoke
    Invoke ..|> ABC

Summary

The invoke.py file provides a robust, configurable HTTP request component for the InfiniFlow framework, supporting dynamic inputs, error handling, proxying, and flexible data posting formats. Its design enables easy integration into complex workflows requiring external API interactions or web crawling functionality with optional HTML content parsing.