llm.go

Overview

The llm.go file defines core interfaces and data structures for interacting with Large Language Models (LLMs) within the system. It provides an abstraction layer that encapsulates LLM invocation, request formulation, and response handling. This file serves as a foundational component in the LLM Integration and Agents topic, enabling higher-level agents and workflows to communicate with underlying LLM providers in a consistent manner.

Key elements of this file include:

The LLM interface that standardizes how different LLM implementations expose their generation capabilities.
The LLMRequest struct representing the input parameters and context for an LLM content generation call.
The LLMResponse struct encapsulating the output, metadata, and streaming state from an LLM generation.

This abstraction allows seamless integration with various LLM backends, supports streaming responses, and provides hooks for metadata like citations, usage, and grounding.

Types and Interfaces

`LLM` Interface

type LLM interface {
    Name() string
    GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error]
}

Purpose: Defines the contract for an LLM implementation that can generate content given a request.
Methods:
- Name() string: Returns the model or implementation name. Useful for logging or diagnostics.
- GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error]: Initiates content generation.
  - Parameters:
    - ctx: Context for cancellation, deadlines, and tracing.
    - req: A pointer to an LLMRequest struct containing the input parameters.
    - stream: Boolean flag indicating whether to stream partial results or return a complete response.
  - Returns: An iterable sequence (iter.Seq2) of LLMResponse pointers or errors. The sequence may yield multiple partial responses when streaming is enabled.
Usage Example:

var llmClient LLM = /* some concrete implementation */
ctx := context.Background()
request := &LLMRequest{
    Model: "gpt-4",
    Contents: []*genai.Content{...},
    Config: &genai.GenerateContentConfig{...},
}
responseSeq := llmClient.GenerateContent(ctx, request, true)

for {
    response, err := responseSeq.Next()
    if err != nil {
        log.Fatal(err)
    }
    if response == nil {
        break
    }
    fmt.Println("Received content:", response.Content)
    if response.TurnComplete {
        break
    }
}

`LLMRequest` Struct

type LLMRequest struct {
    Model    string
    Contents []*genai.Content
    Config   *genai.GenerateContentConfig
    Tools    map[string]any `json:"-"`
}

Purpose: Encapsulates all parameters required to request content generation from an LLM.
Fields:
- Model: The name or identifier of the LLM model to use (e.g., "gpt-4").
- Contents: A slice of pointers to genai.Content objects representing the prompt or input messages for the LLM.
- Config: Pointer to a configuration struct (genai.GenerateContentConfig) that controls generation parameters such as temperature, max tokens, stop sequences, etc.
- Tools: A map for injecting auxiliary tools or resources keyed by string identifiers; this field is ignored in JSON serialization and is intended for internal runtime use.
Notes:
- The Contents field supports flexible prompt composition.
- Tools enables the integration of external utilities or plugins during generation.

`LLMResponse` Struct

type LLMResponse struct {
    Content           *genai.Content
    CitationMetadata  *genai.CitationMetadata
    GroundingMetadata *genai.GroundingMetadata
    UsageMetadata     *genai.GenerateContentResponseUsageMetadata
    CustomMetadata    map[string]any
    LogprobsResult    *genai.LogprobsResult
    Partial           bool
    TurnComplete      bool
    Interrupted       bool
    ErrorCode         string
    ErrorMessage      string
    FinishReason      genai.FinishReason
    AvgLogprobs       float64
}

Purpose: Represents the response from an LLM generation call, including content, metadata, and streaming state.
Fields:
- Content: The generated content as a genai.Content pointer; typically the main text or message generated.
- CitationMetadata: Metadata about citations included in the response (e.g., sources referenced).
- GroundingMetadata: Metadata about grounding information supporting the response.
- UsageMetadata: Contains usage statistics such as token counts and cost estimates.
- CustomMetadata: A flexible map for any additional metadata that may be attached by specific LLM implementations or extensions.
- LogprobsResult: Provides log probability information for tokens if requested, useful for uncertainty or debugging.
- Partial: Indicates if this response is a partial (unfinished) streaming chunk. Only relevant when streaming mode is enabled and content is plain text.
- TurnComplete: Indicates if the current generation turn is complete in streaming scenarios.
- Interrupted: Flags if the generation was interrupted (e.g., due to user cancellation).
- ErrorCode: String code representing an error if one occurred.
- ErrorMessage: Human-readable error message.
- FinishReason: Enumerated reason why generation finished (e.g., stop sequence reached, max tokens).
- AvgLogprobs: Average log probabilities across tokens for quality assessment.
Usage Notes:
- When streaming is off, Partial will be false and a single LLMResponse is returned.
- When streaming is on, multiple LLMResponse objects may be returned with Partial true until TurnComplete becomes true.
- Error fields may be populated if generation failed or was interrupted.

Implementation Details

The file imports context for managing cancellation and deadlines during LLM calls, and iter for representing potentially streaming sequences of responses.
It leverages types from the genai package, which provides standard content structures, configuration, metadata, and enumerations related to generative AI.
The LLM interface abstracts any underlying LLM provider, facilitating polymorphism and ease of integration.
The streaming response model allows consumers to process content as it is generated, enabling real-time user experiences.
Metadata fields support observability, traceability, and enhanced user interaction features such as displaying sources or usage statistics.
The Tools map in LLMRequest facilitates advanced scenarios where auxiliary tools or plugins augment generation, but it is ignored during JSON serialization to prevent leakage in API calls.

Interactions with Other System Components

This file defines core abstractions consumed by agents defined in the LLM Integration and Agents topic. Agents use these interfaces to delegate text generation requests.
The genai package types used here are shared across other components managing AI content and configuration.
The iterable sequence returned by GenerateContent is compatible with the system's event-driven or streaming processing workflows, aligning with Agent Invocation Context and Agent Lifecycle and Callbacks patterns.
The metadata fields in LLMResponse enable telemetry and observability features covered in Telemetry and Observability.
The Tools mechanism in LLMRequest ties into the broader Tooling System, allowing LLMs to utilize external tools or plugins dynamically.
This file acts as a foundational layer for components implementing session management (Session Management) and agent execution runners (Agent Execution Runner) that rely on LLM-generated content.

Diagram: Structure of `llm.go`

classDiagram
class LLM {
+Name() string
+GenerateContent(ctx, req, stream) Seq2[LLMResponse, error]
}
class LLMRequest {
+Model: string
+Contents: []*Content
+Config: *GenerateContentConfig
+Tools: map[string]any
}
class LLMResponse {
+Content: *Content
+CitationMetadata: *CitationMetadata
+GroundingMetadata: *GroundingMetadata
+UsageMetadata: *GenerateContentResponseUsageMetadata
+CustomMetadata: map[string]any
+LogprobsResult: *LogprobsResult
+Partial: bool
+TurnComplete: bool
+Interrupted: bool
+ErrorCode: string
+ErrorMessage: string
+FinishReason: FinishReason
+AvgLogprobs: float64
}
LLM <|.. LLMRequest : uses
LLM ..> LLMResponse : returns sequence of

This documentation covers the purpose, types, methods, and integration points of the llm.go file, providing a detailed reference for developers working with large language model interfaces in the system. For usage in agent construction and orchestration, refer to the related topics on LLM Integration and Agents and Tooling System.

llm.go

Overview

Types and Interfaces

LLM Interface

LLMRequest Struct