For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Distribute requests across the endpoints of a backend with the Power of Two Choices (P2C) algorithm, for all traffic that the gateway proxies.

Distribute requests across the endpoints of a backend, for every kind of traffic that the gateway proxies.

About load balancing

Agentgateway load balances requests across the endpoints of a backend by using the Power of Two Choices (P2C) algorithm with health-aware scoring.

P2C applies to all traffic that the gateway proxies, not only to Large Language Model (LLM) traffic. Plain HTTP, gRPC, Model Context Protocol (MCP), and LLM backends share the same load balancer. The LLM load balancing guide applies the same algorithm across LLM providers.

For each request, agentgateway does the following:

1. Selects two endpoints at random from the highest-priority locality bucket that has viable endpoints. The same endpoint can be selected twice, which keeps the lowest-scored endpoint reachable instead of starving it of traffic.
2. Scores each of the two endpoints based on health, latency, and pending requests.
3. Routes the request to the endpoint with the better score.

What the proxy load balances across

Whether agentgateway load balances across individual pods depends on how you reference the backend, not on the kind of traffic that you send.

By default when you route to a Kubernetes Service, agentgateway connects to a pod IP address directly, not to the ClusterIP, so kube-proxy does not make the load balancing decision.

How endpoints are scored

Agentgateway scores each endpoint on the following factors.

The score for each endpoint is calculated as follows.

score = health / (1 + latency_penalty)
where latency_penalty = request_latency * (1 + pending_requests * 0.1)

Endpoints that consistently fail are moved to a rejected set and are considered only when no active endpoint is viable. As a result, traffic shifts away from a slow or failing pod without any configuration on your part.

Session affinity

P2C selects an endpoint independently for each request, so two requests from the same client can land on different endpoints. To send every request that carries the same value to the same endpoint, set the sessionAffinity backend policy.

A source CEL expression selects the value. Agentgateway hashes the value and maps the hash to an endpoint with weighted rendezvous hashing, so proxy replicas that see the same value and the same set of eligible endpoints pick the same endpoint without sharing any state with each other.

Set the policy on the AgentgatewayBackend that you route to.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: my-backend
  namespace: agentgateway-system
spec:
  policies:
    sessionAffinity:
      source: request.headers["x-session-id"]

You can also set the policy in an AgentgatewayPolicy resource, in the spec.backend section, to apply it to a group of backends.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: my-affinity-policy
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: agentgateway.dev
    kind: AgentgatewayBackend
    name: my-backend
  backend:
    sessionAffinity:
      source: request.headers["x-session-id"]

Common expressions for source include the following.

For the full list of values that an expression can read, see the CEL variables reference.

AI backends

On an AI backend, affinity applies across the provider groups of the backend. Set it on the whole AgentgatewayBackend, not on an individual provider. An AgentgatewayPolicy that sets backend.sessionAffinity and also names a sectionName on an AgentgatewayBackend target is rejected with the message backend.sessionAffinity must target the whole AgentgatewayBackend, not an individual AI provider. The rule applies to any sectioned AgentgatewayBackend target, not only to AI providers.

What session affinity does not do

Session affinity is best-effort, and it is not session persistence. Agentgateway does not record which endpoint a value was sent to. It recomputes the mapping for each request from the value and the set of healthy endpoints, which has two consequences.

• The mapping moves when the endpoint set changes. Adding, removing, or losing an endpoint remaps some values, so a client can be moved to a different endpoint mid-session. Rendezvous hashing keeps that disruption small, because only the values that mapped to the changed endpoint move, but it is not zero.
• A request that produces no usable value is not pinned. Agentgateway falls back to normal P2C selection when the expression fails to evaluate, returns a value that is not a string or bytes, or returns an empty value, such as a header that the client did not send. The request still succeeds.
• Affinity does not override priority or health. The hash chooses among the endpoints that are already eligible, rather than reaching past them. Agentgateway works through the locality priority buckets in order and stops at the first bucket that holds a viable endpoint, and it considers the rejected set only when no active endpoint is viable anywhere. So two replicas in different localities can map the same value to different endpoints, and a value is remapped when the preferred bucket for a replica changes. For more information, see locality-aware routing.

Do not use session affinity to hold server-side state that only one endpoint has. Use it to improve cache hit rates, to keep a conversation on one replica when that is a preference rather than a requirement, or to make debugging easier.

Two other features choose an endpoint before affinity does, and they win when they apply: inference routing, and a stateful MCP session that is already pinned to an upstream server. In practice they do not conflict, because they target different backends.

How load balancing interacts with other routing features

Endpoint selection is one stage of routing. The following features run before or instead of the P2C selection.

Verify which endpoints the proxy resolved

To confirm that the proxy resolved a Service to its endpoints, list the backends that the proxy is routing to.

agctl proxy config backends gateway/agentgateway-proxy -n agentgateway-system

The TYPE column tells you which case you are in. A Service row means that the proxy resolved the Kubernetes Service and load balances across its endpoints, one row per endpoint, named in the ENDPOINT column. A Backend row means an AgentgatewayBackend resource, which might be an opaque hostname.

TYPE     NAME       NAMESPACE            ENDPOINT                    HEALTH  REQUESTS  LATENCY
Backend  openai     agentgateway-system  backend                     0.70    1         0.00ms
Service  ext-authz  backend-extauth      ext-authz-7c7596b5f6-tvs28  1.00    4         0.00ms
Service  httpbin    backend-extauth      httpbin-7dc88b5fbc-zqrfn    1.00    2         3.06ms

For more information about these commands, including how to show services that have received no requests, see Inspect the proxy configuration.

Next steps

• Pin the requests that share a value to one endpoint with session affinity.
• Reduce cross-zone traffic with locality-aware routing.
• Remove failing endpoints from the pool with backend health checking.
• Distribute requests across LLM providers with LLM load balancing.