Telemetry & observability

Enable tracing, prompt logging, cost tracking, and metrics for agentgateway on Kubernetes

Enable distributed tracing and metrics collection for agentgateway on Kubernetes using OpenTelemetry and Jaeger. Get prompt logging, cost tracking, and an audit trail: traces include LLM request/response data and token usage for each request.

What you’ll build

In this tutorial, you will:

Set up a local Kubernetes cluster with agentgateway and an LLM backend
Deploy Jaeger for trace collection and visualization
Configure a TrafficPolicy to enable distributed tracing
Send requests and view traces in the Jaeger UI

Before you begin

Make sure you have the following tools installed:

Docker
kubectl
kind
Helm
An OpenAI API key (get one at platform.openai.com)

For detailed installation instructions, see the LLM Gateway tutorial.

Step 1: Create a kind cluster

kind create cluster --name agentgateway

Step 2: Install agentgateway

# Gateway API CRDs
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml

# agentgateway CRDs
helm upgrade -i --create-namespace \
  --namespace agentgateway-system \
  --version v agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds

# Control plane
helm upgrade -i -n agentgateway-system agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
  --version v

Step 3: Create a Gateway

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

Wait for the proxy:

kubectl get deployment agentgateway-proxy -n agentgateway-system

Step 4: Deploy Jaeger

Deploy Jaeger as a trace collector and visualization tool in its own namespace.

kubectl create namespace telemetry

kubectl apply -n telemetry -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:latest
        ports:
        - containerPort: 16686
          name: ui
        - containerPort: 4317
          name: otlp-grpc
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
spec:
  selector:
    app: jaeger
  ports:
  - port: 16686
    targetPort: 16686
    name: ui
  - port: 4317
    targetPort: 4317
    name: otlp-grpc
EOF

Wait for Jaeger to be ready:

kubectl get pods -n telemetry -w

Step 5: Configure tracing with a TrafficPolicy

Create a TrafficPolicy that sends traces to the Jaeger collector.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TrafficPolicy
metadata:
  name: tracing
  namespace: agentgateway-system
spec:
  targetRefs:
    - kind: Gateway
      name: agentgateway-proxy
      group: gateway.networking.k8s.io
  frontend:
    tracing:
      backendRef:
        name: jaeger
        namespace: telemetry
        port: 4317
      protocol: GRPC
      randomSampling: "true"
EOF

This policy:

Targets the Gateway to apply tracing to all routes
Sends traces to Jaeger via OTLP gRPC on port 4317
Samples all requests (randomSampling: "true") for development

Step 6: Set up an LLM backend

export OPENAI_API_KEY=<insert your API key>

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: openai-secret
  namespace: agentgateway-system
type: Opaque
stringData:
  Authorization: $OPENAI_API_KEY
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        model: gpt-4.1-nano
  policies:
    auth:
      secretRef:
        name: openai-secret
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - backendRefs:
      - name: openai
        namespace: agentgateway-system
        group: agentgateway.dev
        kind: AgentgatewayBackend
EOF

Step 7: Generate traces

Set up port-forwarding for the agentgateway proxy:

kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80 &

Send a few requests to generate traces:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-nano",
    "messages": [{"role": "user", "content": "What is OpenTelemetry?"}]
  }' | jq

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-nano",
    "messages": [{"role": "user", "content": "What is distributed tracing?"}]
  }' | jq

Step 8: View traces in Jaeger

Set up port-forwarding for the Jaeger UI:

kubectl port-forward -n telemetry svc/jaeger 16686:16686 &

Open the Jaeger UI at http://localhost:16686.

Select agentgateway from the Service dropdown
Click Find Traces
Click on a trace to see the full request flow

You’ll see spans for:

The incoming HTTP request
LLM provider routing
Backend request to OpenAI
Response processing

Each span includes details like:

Request and response token counts
Model information
Latency breakdown

Production sampling

For production, use ratio-based sampling to reduce overhead:

frontend:
  tracing:
    backendRef:
      name: otel-collector
      namespace: telemetry
      port: 4317
    protocol: GRPC
    randomSampling: "0.1"  # Sample 10% of traces

Cleanup

kill %1 %2 2>/dev/null
kind delete cluster --name agentgateway

Next steps

Observability Reference

Complete observability configuration

OTel Stack

Full OpenTelemetry stack setup

JWT Authorization

Add security to your deployment

Was this page helpful?