For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Issue API keys with per-key token budgets and cost tracking (also known as virtual keys).

Keep provider API keys on the gateway and give each client a revocable virtual key. Add token budgets and usage tracking as needed.

About

Clients use virtual keys to authenticate to agentgateway. The proxy replaces each virtual key with the provider key stored in a Kubernetes Secret before forwarding the request. Clients never receive the provider key, and the provider never receives the virtual key.

Each client has its own key, so you can revoke one client’s access without rotating the provider key or reconfiguring other clients. Provider-key rotation happens in the cluster.

How virtual keys work

    sequenceDiagram
    participant C as Client<br/>(holds a virtual key)
    participant AGW as Agentgateway<br/>(holds the provider key)
    participant P as LLM provider
    C->>AGW: Request with the virtual key
    AGW->>AGW: Validate the key and check model access
    AGW->>AGW: Replace the virtual key with the provider key
    AGW->>P: Request with the provider key
    P-->>AGW: Response
    AGW-->>C: Response
  

Security considerations

All clients depend on the provider key in the Secret. Protect it as you would any other production credential.

• Restrict who can read the Secret. Limit the Kubernetes RBAC permissions to read Secrets in the namespace of the AgentgatewayBackend.
• Encrypt client traffic. Virtual keys are bearer credentials. Serve clients over HTTPS, or keep the Gateway on a private network.
• Keep agentgateway up to date with security patches.
• Multiple policies: Keep authentication, model authorization, and rate limits in the same policy when they target the same Gateway. Separate policies can overwrite one another even when both report ACCEPTED/ATTACHED.

Before you begin

1. Set up an agentgateway proxy.
2. Set up access to the OpenAI LLM provider.

The OpenAI provider guide stores the provider key in openai-secret. For other ways to provide it, see Provider API keys.

Set up virtual keys

Create virtual keys for Alice and Bob, configure authentication, and verify their access. This setup does not require a rate limit server. Complete the following steps in order.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai: {}
        # Optional: specify a default  model
        #model: gpt-3.5-turbo
     # Optional: custom host and port, if needed
     # host: api.openai.com  
     # port: 443
  policies:
    auth:
      secretRef:
        name: openai-secret
    ai:
      routes:
        "/v1/responses": "Responses"
        "/v1/chat/completions": "Completions"
        "/v1/models": "Models"          # also silences Codex's model-metadata probe
        "*": "Passthrough"
EOF

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - backendRefs:
        - name: openai
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend
EOF

Manage client access

Choose a task to restrict model access, revoke a client key, or rotate provider credentials.

Rotate a provider key

Replace the provider credential in the cluster. Clients keep their virtual keys.

1. Follow Rotate a provider key to replace the credential and revoke the old provider key.
2. Send a request with an existing virtual key. Verify that it still succeeds.

Configure token budgets

Token budgets are optional. Continue here to give Alice and Bob independent daily budgets, shared across proxy replicas. This example uses only 100 tokens per day so that you can test enforcement quickly.

Rate limiting runs before prompt guards, so requests rejected by those guards can still consume quota. Requests rejected during authentication do not consume a token budget.

Configure per-key token budgets

Update the api-key-auth AgentgatewayPolicy from the previous step to also enforce a per-user token budget.

The policy sends a per-user token cost to the rate limit server. It extracts the user_id from each API key and reports the token usage of each response under that descriptor. The rate limit server holds the actual budget (100 tokens per day per user), which you deploy in the next step.

kubectl patch AgentgatewayPolicy api-key-auth -n agentgateway-system \
  --type=merge --patch-file=/dev/stdin <<'EOF'
spec:
  traffic:
    rateLimit:
      global:
        domain: agentgateway
        backendRef:
          kind: Service
          name: ratelimit
          namespace: ratelimit
          port: 8081
        descriptors:
          - entries:
              - name: user_id
                expression: 'apiKey.user_id'
            unit: Tokens
EOF

The merge patch adds the rate limit and keeps the authentication, model restrictions, and metrics settings that the policy already has.

Review the following table to understand this configuration.

Deploy the rate limit server

Global rate limiting requires an external rate limit server that stores the budgets and maintains the counters. Deploy Redis and the rate limit service as described in Deploy the rate limit service in the global rate limiting guide. That example deploys a ratelimit Service in the ratelimit namespace (the target of the backendRef in the previous step) and configures it with the user_id token-budget descriptor that this guide relies on:

# Excerpt from the rate limit server ConfigMap
domain: agentgateway
descriptors:
  - key: user_id
    rate_limit:
      unit: day
      requests_per_unit: 100   # 100 tokens per day per user

The key (user_id) matches the descriptor name in your token budget policy, and the domain (agentgateway) matches the policy’s domain. The requests_per_unit value is the per-user token budget, because the policy reports token usage with unit: Tokens. To change the budget, edit requests_per_unit in the server config; to change the window, edit unit (second, minute, hour, or day).

Verify budget enforcement

1. Send requests with Alice’s API key until her 100-token daily budget is exhausted. Because the LLM provider returns roughly 20-30 tokens per response, a handful of requests pushes Alice over the budget. The request that crosses the budget still completes; subsequent requests are rejected with a 429 status code.

   for i in $(seq 1 10); do
     STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
       "$INGRESS_GW_ADDRESS/v1/chat/completions" \
       -H "Authorization: Bearer vk-alice-4f9c2a" \
       -H "Content-Type: application/json" \
       -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}]}')
     echo "Request $i: HTTP $STATUS"
   done

   Example 429 response:

   HTTP/1.1 429 Too Many Requests
   x-ratelimit-limit: 100
   x-ratelimit-remaining: 0
   x-ratelimit-reset: 43200
   
   rate limit exceeded

2. Verify that Bob can still send requests with his own budget, independent of Alice’s usage.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl "$INGRESS_GW_ADDRESS/v1/chat/completions" \
     -H "Authorization: Bearer vk-bob-8d1e7b" \
      -H "Content-Type: application/json" \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [{"role": "user", "content": "Hello!"}]
     }'

   curl "localhost:8080/v1/chat/completions" \
     -H "Authorization: Bearer vk-bob-8d1e7b" \
      -H "Content-Type: application/json" \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [{"role": "user", "content": "Hello!"}]
     }'

   Bob’s requests succeed because he has his own independent budget.

Monitor per-key spending

Usage tracking is optional and does not require a token budget. Add per-user labels to Prometheus metrics to attribute token usage and estimate spending.

By default, the agentgateway token usage metric (agentgateway_gen_ai_client_token_usage) is broken down by dimensions such as the model and token type, but not by user. To attribute usage to each virtual key, add a user_id label to the metrics with a metrics policy, then query Prometheus.

Before you begin

Set up a Prometheus instance to scrape agentgateway metrics, such as by completing the OpenTelemetry stack guide. The following steps assume the kube-prometheus-stack release exists in the telemetry namespace, as deployed by that guide.

Add a per-key metric label

You can add a per-key metric label such as to track metrics by user ID. Note that the user_id label is high cardinality: every unique value creates a new metric series, which increases Prometheus memory and storage. This is acceptable for tens or hundreds of keys, but avoid attaching unbounded identifiers at large scale. Prefer lower-cardinality dimensions like tier or team when possible.

1. Update the existing api-key-auth policy to add the key’s user_id as a Prometheus label. The merge patch keeps authentication, model restrictions, and any budgets already configured on the Gateway.

   kubectl patch AgentgatewayPolicy api-key-auth -n agentgateway-system \
     --type=merge --patch-file=/dev/stdin <<'EOF'
   spec:
     frontend:
       metrics:
         attributes:
           add:
             - name: user_id
               expression: 'apiKey.user_id'
   EOF

   Review the following table to understand this configuration.

   Setting	Description
   frontend.metrics.attributes.add[].name	The name of the Prometheus label to add (user_id).
   frontend.metrics.attributes.add[].expression	A CEL expression that is evaluated per request. Use apiKey.user_id to read the user_id from the authenticated API key. If the expression fails to evaluate (for example, on an unauthenticated request), the label value is set to unknown.

2. Send a few requests with each virtual key so that the metrics have per-user data to report. You can reuse the requests from Test the virtual keys.

Query per-key usage

1. Port-forward the Prometheus server from the OpenTelemetry stack.

   kubectl port-forward -n telemetry svc/kube-prometheus-stack-prometheus 9090:9090

   Then open the Prometheus UI at http://localhost:9090/graph and run the following queries, or send them to the HTTP API with curl. For example:

   curl -s http://localhost:9090/api/v1/query \
     --data-urlencode 'query=sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum)'

   Example output:

   {"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1782410561.391,"720"]},{"metric":{"user_id":"bob"},"value":[1782410561.391,"448"]},{"metric":{"user_id":"alice"},"value":[1782410561.391,"448"]}]}}

2. Query token usage broken down by user ID. The token usage metric carries a separate series per token type (input, output, input_cache_read), so match both the input and output types in a single selector and sum them, rather than adding two selectors together.

   Querycurl example

   # Total tokens consumed by user over the last 24 hours
   sum by (user_id) (
     increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output"}[24h])
   )
   
   # Percentage of a 100-token daily budget used (adjust the divisor to match your budget)
   (sum by (user_id) (
     increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output"}[24h])
   ) / 100) * 100

   curl -s http://localhost:9090/api/v1/query \
     --data-urlencode 'query=sum by (user_id) (increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output"}[24h]))'

   curl -s http://localhost:9090/api/v1/query \
     --data-urlencode 'query=(sum by (user_id) (increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output"}[24h])) / 100) * 100'

   Each result series is labeled with a user_id, such as alice and bob. If a key is missing the user_id field, or the request is not attributed to a key, its usage appears under user_id="unknown".

   Example output:

   {"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1782411002.488,"0"]},{"metric":{"user_id":"bob"},"value":[1782411002.488,"372.2787929364588"]},{"metric":{"user_id":"alice"},"value":[1782411002.488,"309.56920815395927"]}]}}
   
   {"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1782411059.867,"0"]},{"metric":{"user_id":"bob"},"value":[1782411059.867,"370.95800165527817"]},{"metric":{"user_id":"alice"},"value":[1782411059.867,"307.9427844448483"]}]}}

   Note

   increase() and rate() need at least two samples within the time range to report a value, so a brand-new user_id series shows no result until it has been scraped a few times under continuous traffic. For a quick instant check, query the cumulative counter directly: sum by (user_id) (agentgateway_gen_ai_client_token_usage_sum).

3. Calculate costs per user by multiplying token counts by your provider’s pricing. Input and output tokens are usually priced differently, so reduce each token type to a per-user series with sum by (user_id) before adding them, which keeps the two sides matchable.

   Querycurl example

   # Cost per user (assuming $0.50 per 1M input tokens, $1.50 per 1M output tokens)
   sum by (user_id) (rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h])) / 1000000 * 0.50
   +
   sum by (user_id) (rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h])) / 1000000 * 1.50

   curl -s http://localhost:9090/api/v1/query \
     --data-urlencode 'query=sum by (user_id) (rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h])) / 1000000 * 0.50 + sum by (user_id) (rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h])) / 1000000 * 1.50'

   Example output:

   {"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1782410758.432,"0"]},{"metric":{"user_id":"bob"},"value":[1782410758.432,"6.101636101191084e-09"]},{"metric":{"user_id":"alice"},"value":[1782410758.432,"5.106526900820178e-09"]}]}}

For more information on cost tracking, see the cost tracking guide.

Advanced configuration

For tiered, hourly, and multi-tenant token budgets, see Advanced per-key budgets.

Cleanup

kubectl delete AgentgatewayPolicy api-key-auth -n agentgateway-system --ignore-not-found
kubectl delete configmap llm-api-keys -n agentgateway-system --ignore-not-found
kubectl delete secret llm-api-keys -n agentgateway-system --ignore-not-found
kubectl delete httproute openai -n agentgateway-system
kubectl delete AgentgatewayBackend openai -n agentgateway-system

To remove the rate limit server, follow the cleanup steps in the global rate limiting guide.