llmauditor.go

Overview

This file implements an LLM Auditor Agent that evaluates answers generated by large language models (LLMs) through a two-step process involving critique and revision. It provides a structured framework for auditing LLM outputs by leveraging specialized sub-agents:

1. Critic Agent: Reviews a question-answer pair, identifies claims, verifies their accuracy or reliability, and produces a detailed assessment with justifications and references.

2. Reviser Agent: Takes the critic's findings and minimally revises the original answer text to correct inaccuracies or improve balance while preserving style and length.

The file defines the core prompt templates, callback functions to process model responses, and a factory function to instantiate the auditor agent as a sequential composition of the critic and reviser sub-agents.

This agent is designed to be integrated into workflows requiring high-trust verification of LLM-generated content, particularly in journalistic or editorial contexts where factual accuracy and transparency are critical.

Constants

• EndMark:
  A string marker "---END-OF-EDIT---" used to signify the end of the revised content output by the reviser agent.

• CriticPrompt:
  A detailed multi-step instruction prompt given to the critic agent. It guides the agent to:

  • Identify all claims in an answer.

  • Verify each claim's reliability with possible verdicts (Accurate, Inaccurate, Disputed, Unsupported, Not Applicable).

  • Provide justifications referencing external sources or reasoning.

  • Produce an overall assessment of the answer.

  • Format the output in a specified structured manner.

• ReviserPrompt:
  Instructions for the reviser agent that describe how to minimally modify the original answer based on the critic's findings, preserving style and length, and handling claim verdicts appropriately.
  The reviser outputs either the original or a revised answer, ending with the EndMark.

Functions

afterCritic

func afterCritic(ctx agent.CallbackContext, llmResponse *model.LLMResponse, llmResponseError error) (*model.LLMResponse, error)

• Purpose: Post-processing callback for the critic agent's LLM response.

• Functionality:

  • Checks if the response and its grounding metadata are valid.

  • Extracts references from grounding metadata chunks, including titles, URIs, and text snippets, assembling them into a formatted reference list.

  • Appends the references as a new text part to the LLM response content.

  • Flattens the content parts by joining all text parts into a single text part for simpler consumption downstream.

• Parameters:

  • ctx: The agent invocation context.

  • llmResponse: The response object from the LLM containing content and grounding metadata.

  • llmResponseError: Error from the LLM call, if any.

• Returns:

  • Modified llmResponse with appended references and normalized content parts.

  • Passes through any errors without modification.

• Usage: Automatically invoked after the critic agent produces output to enrich it with structured references for transparency.

afterReviser

func afterReviser(ctx agent.CallbackContext, llmResponse *model.LLMResponse, llmResponseError error) (*model.LLMResponse, error)

• Purpose: Post-processing callback for the reviser agent's LLM response.

• Functionality:

  • Scans the content parts for the EndMark.

  • Truncates all content parts after the first occurrence of the EndMark, removing any trailing text beyond it.

  • Ensures the final revised answer does not include the marker in the output.

• Parameters:

  • ctx: The agent callback context.

  • llmResponse: The LLM response with content parts.

  • llmResponseError: Error from the LLM call, if any.

• Returns:

  • Cleaned-up llmResponse with content truncated at the end marker.

  • Passes through any errors without modification.

• Usage: Ensures the reviser's output is cleanly formatted for downstream consumption.

GetLLMAuditorAgent

func GetLLMAuditorAgent(ctx context.Context, model model.LLM) agent.Agent

• Purpose: Factory function to create a composite LLM Auditor agent.

• Functionality:

  • Instantiates two LLM agents (criticAgent and reviserAgent) using the provided LLM model and respective prompt instructions (CriticPrompt and ReviserPrompt).

  • Assigns the appropriate post-processing callbacks (afterCritic, afterReviser) to each agent.

  • Composes these two sub-agents into a sequential agent (rootAgent) that runs the critic first, then the reviser.

  • Configures the composite agent with a descriptive name and sub-agent references.

  • Panics on errors during agent creation (ensures fail-fast for configuration issues).

• Parameters:

  • ctx: Context for agent creation and cancellation.

  • model: The underlying LLM model interface used by both sub-agents.

• Returns:

  • A sequentialagent that orchestrates the auditing workflow combining critique and revision.

• Usage: Entry point to obtain a ready-to-use LLM Auditor agent for evaluation pipelines.

Implementation Details and Algorithms

• Claim Verification Workflow:
  The auditor uses static prompt templates (CriticPrompt, ReviserPrompt) that implement a multi-step verification workflow. This manual prompt design encodes best practices for claim extraction, verification, justification, and minimal revision, simulating a professional editorial process.

• Grounding Metadata Handling:
  The afterCritic callback enriches the critic's output by extracting grounding metadata (such as retrieved documents or web snippets) and appending a human-readable reference section. This improves transparency and traceability of the verification.

• Sequential Agent Composition:
  The auditor is modeled as a sequentialagent running two distinct sub-agents in sequence, reflecting a pipeline architecture: critique first, then revision. This aligns with Agent Workflow Management principles.

• Error Handling:
  The factory panics on constructor errors to ensure early failure detection during initialization rather than runtime errors during audit operations.

Interaction with Other System Components

• LLM Model Interface (model.LLM):
  The auditor depends on an external LLM backend, abstracted via the model.LLM interface, to generate critique and revision outputs.

• Agent Framework (agent.Agent):
  The components created conform to the agent framework, supporting lifecycle, callbacks, and sub-agent composition as described in LLM Integration and Agents and Agent Workflow Management.

• Sequential Agent (sequentialagent):
  The auditor leverages the sequential agent sub-framework to run the critic and reviser agents in order, based on Sequential Agent.

• Callback Context and Post-Processing:
  The post-processing functions afterCritic and afterReviser interact with agent invocation lifecycle hooks defined in Agent Lifecycle and Callbacks.

Usage Example

ctx := context.Background()
model := YourLLMImplementation{} // satisfies model.LLM interface

llmAuditor := GetLLMAuditorAgent(ctx, model)

// Use llmAuditor to evaluate question-answer pairs.
// The agent first critiques the answer, then revises it based on findings.

Mermaid Diagram

classDiagram
class LLMAuditorAgent {
+GetLLMAuditorAgent(ctx, model) Agent
}
class CriticAgent {
+CriticPrompt: string
+afterCritic(ctx, llmResponse, err) (*LLMResponse, error)
}
class ReviserAgent {
+ReviserPrompt: string
+afterReviser(ctx, llmResponse, err) (*LLMResponse, error)
}
class SequentialAgent {
+SubAgents: []Agent
}
LLMAuditorAgent --> SequentialAgent : creates
SequentialAgent --> CriticAgent : runs first
SequentialAgent --> ReviserAgent : runs second
CriticAgent --> afterCritic
ReviserAgent --> afterReviser

This documentation covers all significant aspects of the llmauditor.go file, including purpose, key functions, implementation details, system interactions, and a structural diagram. For details on the agent framework, lifecycle callbacks, and sequential agents, refer to LLM Integration and Agents, Agent Lifecycle and Callbacks, and Sequential Agent.