telemetry.go

Overview

The telemetry.go file in the telemetry package implements the core telemetry and observability functionality for the AI Agent Development Kit (ADK). Its primary purpose is to set up OpenTelemetry tracing infrastructure and emit detailed trace spans for key agent operations such as LLM calls and tool executions. This enables comprehensive visibility into the runtime behavior of AI agents by recording span events enriched with contextual metadata from sessions, agent invocations, and model/tool interactions.

The file manages a local tracer provider configured with span processors, offers APIs to start and end trace spans, and populates span attributes with serialized request/response data. It integrates tightly with agent, session, tool, and model components to correlate telemetry data with operational events.

Types and Variables

tracerProviderHolder

type tracerProviderHolder struct {
	tp trace.TracerProvider
}

• Holds a reference to an OpenTelemetry TracerProvider.

• Used to encapsulate the local tracer provider instance.

tracerProviderConfig

type tracerProviderConfig struct {
	spanProcessors []sdktrace.SpanProcessor
	mu             *sync.RWMutex
}

• Contains configuration for the local tracer provider.

• Maintains a slice of registered span processors.

• Protects concurrent access with a read-write mutex.

Package Variables

• once sync.Once: Ensures RegisterTelemetry runs only once for singleton setup.

• localTracer tracerProviderHolder: Holds the initialized local tracer provider.

• localTracerConfig tracerProviderConfig: Holds span processors and mutex for configuration.

• Constant strings defining attribute keys and system names used in span attributes.

Functions

AddSpanProcessor

func AddSpanProcessor(processor sdktrace.SpanProcessor)

• Adds a span processor to localTracerConfig in a thread-safe manner.

• Parameters:

  • processor: An OpenTelemetry span processor to be registered.

• Usage:

  • Called before telemetry registration to add exporters or processors for trace export.

• Example:

  telemetry.AddSpanProcessor(mySpanProcessor)
  

RegisterTelemetry

func RegisterTelemetry()

• Initializes the local tracer provider exactly once.

• Registers all span processors stored in localTracerConfig to the new tracer provider.

• Sets localTracer with the created provider.

• Ensures thread-safe one-time setup using sync.Once.

• Usage:

  • Called automatically if the local tracer is not yet initialized when starting traces.

getTracers

func getTracers() []trace.Tracer

• Returns a slice of two tracers:

  1. The local tracer from localTracer.tp initialized by RegisterTelemetry.

  2. The global OpenTelemetry tracer from otel.GetTracerProvider().

• If local tracer is uninitialized, calls RegisterTelemetry.

• Enables dual tracing to support both local and global telemetry configurations.

StartTrace

func StartTrace(ctx context.Context, traceName string) []trace.Span

• Starts and returns two spans (one from each tracer returned by getTracers).

• Parameters:

  • ctx: The parent context for the span.

  • traceName: Name for the trace span.

• Returns:

  • Slice of active trace.Span objects.

• Usage:

  • Used to begin tracing an operation (e.g., LLM call or tool execution).

  • Caller is responsible for ending spans.

TraceMergedToolCalls

func TraceMergedToolCalls(spans []trace.Span, fnResponseEvent *session.Event)

• Traces a merged tool execution event.

• Parameters:

  • spans: Slice of spans to set attributes on and end.

  • fnResponseEvent: The session event representing the tool response.

• Sets fixed attributes for merged tool calls (e.g., tool name as "(merged tools)").

• Serializes the response event safely for span attributes.

• Ends all spans after setting attributes.

TraceToolCall

func TraceToolCall(spans []trace.Span, tool tool.Tool, fnArgs map[string]any, fnResponseEvent *session.Event)

• Traces an individual tool execution event.

• Parameters:

  • spans: Spans to annotate and close.

  • tool: The tool instance executed.

  • fnArgs: Arguments passed to the tool.

  • fnResponseEvent: The session event containing the tool's response.

• Extracts tool call ID and response content, safely serialized.

• Sets attributes related to tool name, description, arguments, call ID, and response.

• Ends all spans.

TraceLLMCall

func TraceLLMCall(spans []trace.Span, agentCtx agent.InvocationContext, llmRequest *model.LLMRequest, event *session.Event)

• Traces a large language model (LLM) call event.

• Parameters:

  • spans: Spans to annotate.

  • agentCtx: Invocation context providing session info.

  • llmRequest: The LLM request object.

  • event: The session event with the LLM response.

• Sets attributes including system name, model, invocation/session/event IDs.

• Serializes filtered LLM request and response data.

• Optionally adds attributes for TopP and MaxOutputTokens if set.

• Ends all spans.

safeSerialize

func safeSerialize(obj any) string

• Marshals an object to JSON string safely.

• Returns "<not serializable>" if marshaling fails.

• Used to convert complex request/response objects to string attributes without panics.

llmRequestToTrace

func llmRequestToTrace(llmRequest *model.LLMRequest) map[string]any

• Converts an LLM request into a map suitable for tracing.

• Filters out inline data parts from contents for privacy and relevance.

• Returns a map with keys: "config", "model", and "content" (filtered).

• Used internally by TraceLLMCall to prepare the LLM request data for serialization.

Implementation Details and Algorithms

• Singleton Tracer Provider: The local tracer provider is initialized once using sync.Once to guarantee thread-safe singleton behavior.

• Dual Tracer Model: Both local and global tracers are used for flexibility, allowing telemetry data to be captured in different configured backends.

• Span Attribute Enrichment: Tracing functions enrich spans with detailed metadata relevant to the operation type (LLM call or tool call), including serialized request/response payloads, tool names, descriptions, and unique IDs.

• Safe Serialization: Serialization errors are gracefully handled to avoid telemetry failures.

• Filtering Inline Data: llmRequestToTrace filters out inline data from LLM content parts before tracing, reducing noise and protecting sensitive data.

Interactions With Other System Components

• Agent (agent package): Provides invocation context and session information used for telemetry attributes.

• Session (session package): Supplies event and session IDs that are attached to spans for trace correlation.

• Tool (tool package): Supplies tool metadata (name, description) and execution arguments for tracing tool calls.

• Model (model package): Supplies LLM request and response data structures.

• OpenTelemetry SDK: Core integration with OpenTelemetry APIs for tracer providers, span creation, and span processors.

The telemetry file acts as a centralized observability layer, embedding tracing instrumentation seamlessly into agent runtime operations without requiring manual instrumentation in client code. This enables rich, correlated telemetry data collection for system monitoring and debugging.

Usage Examples

Registering Custom Span Processor

processor := sdktrace.NewSimpleSpanProcessor(myExporter)
telemetry.AddSpanProcessor(processor)
telemetry.RegisterTelemetry()

Starting a Trace for an LLM Request

spans := telemetry.StartTrace(ctx, "gcp.vertex.agent.llm_request")
// Perform LLM call...
telemetry.TraceLLMCall(spans, invocationContext, llmRequest, event)

Tracing a Tool Call

spans := telemetry.StartTrace(ctx, "execute_tool")
// Execute tool...
telemetry.TraceToolCall(spans, toolInstance, toolArgs, responseEvent)

Mermaid Diagram of File Structure and Function Relationships

flowchart TB
AddSpanProcessor --> RegisterTelemetry
RegisterTelemetry --> localTracer
getTracers --> RegisterTelemetry
StartTrace --> getTracers
StartTrace --> trace.Spans
TraceLLMCall --> StartTrace
TraceToolCall --> StartTrace
TraceMergedToolCalls --> StartTrace
safeSerialize --> TraceToolCall
safeSerialize --> TraceMergedToolCalls
safeSerialize --> TraceLLMCall
llmRequestToTrace --> TraceLLMCall

• AddSpanProcessor adds processors before initialization.

• RegisterTelemetry initializes local tracer and registers processors.

• getTracers returns both local and global tracers, initializing local tracer if needed.

• StartTrace starts spans on all tracers.

• Trace functions (TraceLLMCall, TraceToolCall, TraceMergedToolCalls) annotate spans and end them.

• Utility functions safeSerialize and llmRequestToTrace support safe data serialization.

Related Topics

• Telemetry and Observability — Provides context on telemetry concepts and how this file fits into the overall observability strategy.

• OpenTelemetry Integration — Describes span processor registration and instrumentation specific to LLM and tool calls.

• Agent Invocation Context — Defines the agent context used for session and invocation IDs in telemetry.

• Session Management — Provides session and event information utilized in trace attributes.

• Tooling System — Defines the tools whose executions are traced by this file.

• LLM Integration and Agents — Source of LLM request and response data structures used in tracing.

These references offer deeper understanding of the components integrated with telemetry.go.