Inference routing

Route AI inference requests to LLM workloads using the Kubernetes Gateway API Inference Extension.

Use agentgateway with the Kubernetes Gateway API Inference Extension to route requests to Large Language Model (LLM) workloads in your Kubernetes environment.

The Gateway API Inference Extension defines the InferencePool API and the protocol between gateways and endpoint pickers. The llm-d Router provides a production-oriented Endpoint Picker (EPP) implementation. agentgateway routes to the InferencePool, the llm-d Router selects a model server from the pool, and agentgateway routes to that model server endpoint.

For more information, see the following resources.

Gateway API Inference Extension llm-d gateway infrastructure llm-d with agentgateway Standalone inference routing

About

An InferencePool groups model server pods into a routable Gateway API backend. Its endpointPickerRef identifies the llm-d Router EPP that selects a pod for each request.

Agentgateway implements the gateway side of the Inference Extension protocol. The following diagram shows the request flow.

    graph LR
    Client --> Gateway
    Gateway --> HTTPRoute
    HTTPRoute --> InferencePool
    InferencePool --> EPP["llm-d Router EPP"]
    EPP --> ModelServer["model server"]

The EPP returns its selected endpoint to agentgateway. Agentgateway then sends the request to that model server and returns the response to the client.

Quickstart

In this quickstart, you deploy a simulated model server, the Gateway API Inference Extension CRDs, agentgateway, and the llm-d Router in Gateway mode.

Deploy the Qwen3 model server simulator. The llm-d-inference-sim container mimics a vLLM model server without downloading model weights or requiring GPUs.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3-32b
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vllm-qwen3-32b
  template:
    metadata:
      labels:
        app: vllm-qwen3-32b
        inference.networking.k8s.io/engine-type: vllm
    spec:
      containers:
        - name: vllm-sim
          image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2
          args:
            - --model
            - Qwen/Qwen3-32B
            - --port
            - "8000"
            - --max-loras
            - "2"
            - --lora-modules
            - '{"name": "food-review-1"}'
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          resources:
            requests:
              cpu: 10m
EOF

Verify that the simulator deployment is available.

kubectl wait --for=condition=available --timeout=120s \
  deployment/vllm-qwen3-32b

Install the Gateway API and Gateway API Inference Extension Custom Resource Definitions (CRDs).

kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml

kubectl apply -f \
  https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml

Install agentgateway with Inference Extension support.

helm upgrade -i --create-namespace \
  --namespace agentgateway-system \
  --version 0.0.0-latest-dev \
  agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds

helm upgrade -i \
  --namespace agentgateway-system \
  --version 0.0.0-latest-dev \
  --set inferenceExtension.enabled=true \
  agentgateway oci://cr.agentgateway.dev/charts/agentgateway

Create an agentgateway Gateway.

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
  namespace: default
spec:
  gatewayClassName: agentgateway
  listeners:
    - name: http
      port: 80
      protocol: HTTP
EOF

Verify that the Gateway is programmed.

kubectl wait --for=condition=Programmed --timeout=120s \
  gateway/inference-gateway

Install the llm-d Router Gateway chart. The chart creates the InferencePool, the llm-d Router EPP deployment and service, and an HTTPRoute that attaches to the Gateway.

export ROUTER_CHART_VERSION=v0.9.0

helm upgrade -i vllm-qwen3-32b \
  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \
  --version $ROUTER_CHART_VERSION \
  --set router.modelServers.matchLabels.app=vllm-qwen3-32b \
  --set router.epp.resources.requests.cpu=100m \
  --set router.epp.resources.requests.memory=128Mi \
  --set router.epp.resources.limits.memory=512Mi \
  --set provider.name=none \
  --set httpRoute.create=true \
  --set httpRoute.inferenceGatewayName=inference-gateway

The none provider value prevents the chart from creating resources for a different gateway implementation. Agentgateway still processes the chart-created HTTPRoute and InferencePool.

Verify the pool, EPP image, and route.

kubectl get inferencepool vllm-qwen3-32b
kubectl get deployment vllm-qwen3-32b-epp \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl get httproute vllm-qwen3-32b

The deployment uses the released ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0 image.

Send a request through agentgateway.

kubectl port-forward service/inference-gateway 8080:80

In a separate terminal, send the request.

curl -i http://localhost:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-32B",
    "prompt": "What is the warmest city in the USA?",
    "max_tokens": 100,
    "temperature": 0.5
  }'

The response has an HTTP 200 OK status.

Use AI policies with InferencePools

The quickstart routes directly from an HTTPRoute to an InferencePool. Use that pattern when you only need Gateway API Inference Extension endpoint selection.

To also use agentgateway LLM features, such as token counting, token-based rate limits, guardrails, transformations, and LLM observability, route the HTTPRoute to an AgentgatewayBackend. Then, configure a custom provider on the AgentgatewayBackend that targets the InferencePool.

    graph LR
    Client --> Gateway
    Gateway --> HTTPRoute
    HTTPRoute --> AgentgatewayBackend
    AgentgatewayBackend --> InferencePool
    InferencePool --> ModelServer["model server"]

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: qwen-inferencepool
  namespace: default
spec:
  ai:
    provider:
      custom:
        backendRef:
          group: inference.networking.k8s.io
          kind: InferencePool
          name: vllm-qwen3-32b
        model: Qwen/Qwen3-32B
        formats:
        - type: Completions
          path: /v1/chat/completions
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: vllm-qwen3-32b
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/chat/completions
    backendRefs:
    - group: agentgateway.dev
      kind: AgentgatewayBackend
      name: qwen-inferencepool
    timeouts:
      request: 300s

The following example applies an LLM token budget to the same HTTPRoute. Because the route points to an AgentgatewayBackend with a custom provider, agentgateway can parse the LLM response usage and enforce the token limit while still using the InferencePool for endpoint selection.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: qwen-token-budget
  namespace: default
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: vllm-qwen3-32b
  traffic:
    rateLimit:
      local:
      - tokens: 1000
        unit: Minutes

For more token rate limiting details, see Rate limiting for LLMs.

For more custom provider examples, see Custom providers.

To route requests to multiple InferencePools based on the request body model field, see Multiple inference pools.

Note

Most users can keep the default llm-d Router OpenAI parser and send OpenAI-compatible requests, such as /v1/chat/completions. If clients send a different request format, configure the EPP parser, such as router.epp.parser, for that client-facing format. For parser options, see the llm-d Router parser docs.

Multiple inference pools

Was this page helpful?

Inference routing

About

Quickstart

Use AI policies with InferencePools

What could be improved?