For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Serve a model

Expose an LLM model to clients with an AgentgatewayModel resource, including wildcard matching and provider credentials.

Expose an LLM model to clients with an AgentgatewayModel resource.

About

An AgentgatewayModel declares one client-facing model and attaches it to a Gateway listener. Agentgateway derives the LLM routing from the models that attach to a listener, so you do not create an AgentgatewayBackend or an HTTPRoute.

In this guide, you enable the API, turn on LLM serving for a listener, and expose three kinds of models: an exact match, a wildcard match, and a model that authenticates to its provider with a Kubernetes Secret.

For more information, see About models.

Before you begin

1. Set up an agentgateway proxy.

2. Deploy the httpbun mock LLM. This guide routes to httpbun so that you do not need a provider API key. To use a real provider instead, remove the baseURL field from each model and follow API keys.

3. Enable the AgentgatewayModel API on the control plane. The API is experimental and disabled by default, so it is not available in a standard installation.

   helm upgrade -i -n agentgateway-system agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
   --version 0.0.0-latest-dev \
   --reuse-values \
   --set controller.image.pullPolicy=Always \
   --set agentgatewayModels.enabled=true \
   --wait

   This command uses the nightly development build, because the AgentgatewayModel API is not yet in a released chart.

4. Verify that the API is enabled. The command returns true when the feature gate is set.

   kubectl get deploy agentgateway -n agentgateway-system \
     -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="AGW_ENABLE_AGENTGATEWAY_MODELS")].value}'

Enable LLM serving on a listener

A listener serves LLM traffic only when it allows the AgentgatewayModel route kind. In this guide, you add the route kind to the http listener on the agentgateway-proxy Gateway that you created during setup. The gateway then serves both LLM models and ordinary HTTPRoute traffic on the same port.

1. Update the listener to allow both HTTPRoute and AgentgatewayModel resources.

   kubectl apply -f- <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: agentgateway-proxy
     namespace: agentgateway-system
   spec:
     gatewayClassName: agentgateway
     listeners:
     - name: http
       port: 80
       protocol: HTTP
       allowedRoutes:
         namespaces:
           from: All
         kinds:
         - group: gateway.networking.k8s.io
           kind: HTTPRoute
         - group: agentgateway.dev
           kind: AgentgatewayModel
   EOF

   Review the following table to understand this configuration.

   Field	Value	Description
   allowedRoutes.kinds	AgentgatewayModel and HTTPRoute	Adding AgentgatewayModel enables the listener’s built-in LLM paths, such as /v1/chat/completions and /v1/models. Keeping HTTPRoute lets your existing routes continue to work. After you set allowedRoutes.kinds, the listener accepts only the kinds that you list, so include every kind that you need.
   listeners[0].name	http	Models reference this name in parentRefs.sectionName.

2. Verify that the listener now supports both kinds.

   kubectl get gateway agentgateway-proxy -n agentgateway-system \
     -o jsonpath='{.status.listeners[0].supportedKinds}'

   Example output:

   [{"group":"gateway.networking.k8s.io","kind":"HTTPRoute"},{"group":"agentgateway.dev","kind":"AgentgatewayModel"}]

3. Save the gateway address in an environment variable, if you have not already.

   Cloud Provider LoadBalancerPort-forward for local testing

   export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
   echo $INGRESS_GW_ADDRESS

   Use this option if your cluster does not assign an external IP to LoadBalancer services, such as a default Kind cluster.

   kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80

   In a separate terminal, point the requests in this guide at the forwarded port.

   export INGRESS_GW_ADDRESS=localhost:8080

The listener now serves LLM traffic, but it has no models yet, so every LLM request returns a model_not_found error.

Serve an exact model

When you omit spec.match, the model matches metadata.name exactly. Clients request this model as gpt-4.

1. Create the model.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayModel
   metadata:
     name: gpt-4
     namespace: agentgateway-system
   spec:
     parentRefs:
     - group: gateway.networking.k8s.io
       kind: Gateway
       name: agentgateway-proxy
       sectionName: http
     provider: OpenAI
     baseURL: http://httpbun.default.svc.cluster.local:3090/llm
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Field	Value	Description
   parentRefs	Gateway	Attaches the model to the http listener. Omit sectionName to attach to every eligible listener on the Gateway.
   provider	OpenAI	The provider that serves this model. httpbun implements the OpenAI-compatible API.
   baseURL	http://httpbun.default.svc.cluster.local:3090/llm	Overrides the provider address and base path prefix, so requests go to httpbun’s /llm/chat/completions endpoint instead of OpenAI. Remove this field to use the real provider.

2. Send a request for the model.

   curl -X POST http://$INGRESS_GW_ADDRESS/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "gpt-4",
       "messages": [{"role": "user", "content": "Hello!"}],
       "httpbun": {"content": "Hello from the mock LLM"}
     }'

   Example output:

   {"model":"gpt-4","usage":{"prompt_tokens":3,"completion_tokens":4,"total_tokens":7},"choices":[{"message":{"content":"Hello from the mock LLM","role":"assistant"},"finish_reason":"stop","index":0}],"object":"chat.completion"}

3. Request a model that does not exist. Agentgateway returns an OpenAI-compatible error.

   curl -X POST http://$INGRESS_GW_ADDRESS/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "does-not-exist", "messages": []}'

   Example output:

   {"error":{"message":"Model not found","type":"invalid_request_error","code":"model_not_found"}}

Serve a family of models with a wildcard

Use spec.match.model to match more than one model name. The provider does not recognize the client-facing prefix, so pair the wildcard with a model transformation that rewrites the name before the request leaves the gateway.

1. Create a model that matches any name that starts with openai/.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayModel
   metadata:
     name: openai-models
     namespace: agentgateway-system
   spec:
     parentRefs:
     - group: gateway.networking.k8s.io
       kind: Gateway
       name: agentgateway-proxy
       sectionName: http
     match:
       model: openai/*
     provider: OpenAI
     baseURL: http://httpbun.default.svc.cluster.local:3090/llm
     policies:
       transformations:
       - field: model
         expression: 'llmRequest.model.stripPrefix("openai/")'
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Field	Value	Description
   model	openai/*	Matches any model name that starts with openai/. Wildcards must be *, a suffix such as openai/*, or a prefix such as *-latest.
   field	model	The field in the provider request body to rewrite, as part of a transformation policy.
   expression	llmRequest.model.stripPrefix("openai/")	Removes the client-facing prefix, so the provider receives gpt-5-mini rather than openai/gpt-5-mini.

2. Request a model through the wildcard. The response shows the transformed model name.

   curl -X POST http://$INGRESS_GW_ADDRESS/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "openai/gpt-5-mini",
       "messages": [{"role": "user", "content": "Hello!"}],
       "httpbun": {"content": "Hello from the mock LLM"}
     }'

   Example output. Note that model is gpt-5-mini, not openai/gpt-5-mini.

   {"model":"gpt-5-mini","usage":{"prompt_tokens":3,"completion_tokens":4,"total_tokens":7},"choices":[{"message":{"content":"Hello from the mock LLM","role":"assistant"},"finish_reason":"stop","index":0}],"object":"chat.completion"}

Authenticate to the provider

Real providers require credentials. Use spec.policies.auth to read them from a Kubernetes Secret.

1. Create a Secret that holds the provider API key. By default, agentgateway reads the Authorization key.

   kubectl apply -f- <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: openai-secret
     namespace: agentgateway-system
   type: Opaque
   stringData:
     Authorization: $OPENAI_API_KEY
   EOF

2. Create a model that references the Secret.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayModel
   metadata:
     name: gpt-5-mini
     namespace: agentgateway-system
   spec:
     parentRefs:
     - group: gateway.networking.k8s.io
       kind: Gateway
       name: agentgateway-proxy
       sectionName: http
     provider: OpenAI
     policies:
       auth:
         secretRef:
           name: openai-secret
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Field	Value	Description
   secretRef	openai-secret	The Secret to read the credential from. Set secretRef.key to read a key other than Authorization.

   Agentgateway writes the credential to the Authorization header with a Bearer prefix. Use policies.auth.location to write it somewhere else, such as a custom header or query parameter.

Verify model discovery

Every public model on a listener appears in the OpenAI-compatible discovery endpoint. The endpoint requires no configuration.

curl -s http://$INGRESS_GW_ADDRESS/v1/models

Example output:

{
  "data": [
    {"id": "gpt-4", "object": "model", "created": 1785166485, "owned_by": "openai"},
    {"id": "gpt-5-mini", "object": "model", "created": 1785166485, "owned_by": "openai"},
    {"id": "openai/*", "object": "model", "created": 1785166485, "owned_by": "openai"}
  ],
  "object": "list"
}

Wildcard models are listed by their match pattern. Models with visibility: Internal are excluded.

Troubleshooting

Requests return model_not_found

What’s happening:

Every request fails with the following error, even for a model you created.

{"error":{"message":"Model not found","type":"invalid_request_error","code":"model_not_found"}}

Why it’s happening:

The model did not attach to the listener, or it is not reachable by clients. Common causes include the following.

• The AgentgatewayModel API is not enabled on the control plane.
• The listener does not allow the AgentgatewayModel route kind.
• The parentRefs field does not match the Gateway name or sectionName.
• The model is in a different namespace than the Gateway, and the listener restricts allowedRoutes.namespaces.
• The model sets visibility: Internal, so clients cannot request it directly.

How to fix it:

1. Confirm that the API is enabled. If the following command returns no output, repeat the Helm step in Before you begin.

   kubectl get deploy agentgateway -n agentgateway-system \
     -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="AGW_ENABLE_AGENTGATEWAY_MODELS")].value}'

2. Check how many routes attached to the listener. This count includes every AgentgatewayModel that attached, whether public or internal.

   kubectl get gateway agentgateway-proxy -n agentgateway-system \
     -o jsonpath='{.status.listeners[0].attachedRoutes}'

3. List the models that clients can reach. If a model is missing here but the attachedRoutes count includes it, the model is Internal.

   curl -s http://$INGRESS_GW_ADDRESS/v1/models

Cleanup

kubectl delete agentgatewaymodel gpt-4 openai-models gpt-5-mini -n agentgateway-system
kubectl delete secret openai-secret -n agentgateway-system

To stop the listener from serving LLM traffic, remove the AgentgatewayModel entry from allowedRoutes.kinds. Do not delete the agentgateway-proxy Gateway, because other guides use it.