Prometheus

Scrape agentgateway metrics with Prometheus.

Prometheus is an open-source monitoring system that scrapes and stores time-series metrics. It is the standard way to collect and query agentgateway metrics, and is the data source behind the Grafana dashboard. Prometheus is well-suited for agentgateway because it efficiently handles high-cardinality label sets, such as per-route, per-model, and per-provider breakdowns. Its query language (PromQL) makes it easy to build alerts and dashboards from those labels.

Agentgateway is built to work with Prometheus out of the box:

It exposes a /metrics endpoint on port 15020 in the OpenMetrics format, which Prometheus can scrape directly.
All metrics use the agentgateway_ prefix and carry consistent route identifier labels so you can filter and aggregate by gateway, listener, and route without additional configuration.
On Kubernetes, the agentgateway chart annotates proxy pods with prometheus.io/scrape: "true" and prometheus.io/port: "15020" for Prometheus installs that use annotation-based discovery, and ships an optional PodMonitor for installs that run the Prometheus Operator.

Set up Prometheus

Set up a Prometheus instance and use PromQL queries to query them. The steps to set up Prometheus vary depending on the way you installed agentgateway.

Create a Prometheus configuration file that scrapes agentgateway.

cat > prometheus.yml <<EOF
scrape_configs:
  - job_name: agentgateway
    static_configs:
      - targets:
          - host.docker.internal:15020
    scrape_interval: 15s
EOF

Run Prometheus with Docker.

docker run -d --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus:latest

Open the Prometheus UI at http://localhost:9090.
Run a PromQL query to verify that agentgateway metrics are being scraped. In the Prometheus UI, enter the following query and click Execute.
```
agentgateway_requests_total
```
The result shows the total number of requests that are handled by agentgateway, broken down by gateway, listener, route, and status.
When you are done, remove the Prometheus container.
```
docker rm -f prometheus
```

The recommended way to run Prometheus on Kubernetes is with the kube-prometheus-stack Helm chart, which installs Prometheus, Alertmanager, and Grafana together.

The Prometheus Operator that this chart installs discovers scrape targets through PodMonitor and ServiceMonitor resources, not through the prometheus.io/scrape pod annotations. To scrape agentgateway, enable the PodMonitor that the agentgateway chart ships, and label it with the Prometheus release name so that the operator selects it.

Add the Prometheus community Helm repository.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install the kube-prometheus-stack chart.

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Verify that the pods are running.
```
kubectl get pods -n monitoring
```

Enable the agentgateway PodMonitor so that Prometheus scrapes the proxy’s metrics endpoint. The release label must match the name of your kube-prometheus-stack release.

helm upgrade -i agentgateway-standalone \
  oci://cr.agentgateway.dev/charts/agentgateway-standalone \
  --namespace agentgateway-system \
  --version 0.0.0-latest-dev \
  --reuse-values \
  --set monitoring.enabled=true \
  --set monitoring.extraLabels.release=kube-prometheus-stack

Verify that the PodMonitor was created.

kubectl get podmonitor -n agentgateway-system

Port-forward the Prometheus UI service.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090

Open the Prometheus UI http://localhost:9090 to query metrics.
Run a PromQL query to verify that agentgateway metrics are being scraped. In the Prometheus UI, enter the following query and click Execute.
```
agentgateway_requests_total
```
The result shows the total number of requests handled by agentgateway, broken down by gateway, listener, route, and status.

When you are done, remove the stack and the namespace.

helm uninstall kube-prometheus-stack -n monitoring
kubectl delete namespace monitoring

Other configurations

Review other common Prometheus configurations.

Docker Compose

To run agentgateway, Prometheus, and Grafana together as a single stack, use Docker Compose instead of individual containers.

services:
  agentgateway:
    image: cr.agentgateway.dev/agentgateway:latest
    ports:
      - "3000:3000"
      - "15020:15020"
    volumes:
      - ./config.yaml:/config.yaml:ro
    command: ["-f", "/config.yaml"]

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  grafana-data:

Learn more

Grafana dashboardVisualize agentgateway metrics with the pre-built Grafana dashboard Metrics referenceFull list of agentgateway metrics and labels

View and customize metrics Grafana

Was this page helpful?

Prometheus

Set up Prometheus

Other configurations

Docker Compose

Learn more

What could be improved?