LLM Observability

Agent Gateway can send LLM telemetry to specialized observability platforms for prompt analytics, cost tracking, and performance monitoring.

How it works

Agent Gateway exports LLM telemetry via OpenTelemetry, which can be forwarded to LLM-specific observability platforms. These platforms provide:

• Prompt/response logging - Full request and response capture
• Token usage tracking - Monitor costs across models and users
• Latency analytics - Track response times and identify bottlenecks
• Evaluation - Score and evaluate LLM outputs
• Prompt management - Version and manage prompts

Configuration

Enable OpenTelemetry tracing with LLM-specific attributes:

config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: true

binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - ai:
          name: openai
          provider:
            openAI:
              model: gpt-4o-mini
      policies:
        backendAuth:
          key: "$OPENAI_API_KEY"

Agent Gateway automatically includes these LLM-specific trace attributes: