For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

About models

Learn how the AgentgatewayModel API provides a model-centric way to serve LLMs in Kubernetes.

Learn how the AgentgatewayModel API provides a model-centric way to serve LLMs in Kubernetes.

About

Clients talk to an LLM gateway in terms of models. They send a request with "model": "gpt-5-mini" in the body and expect the gateway to figure out the rest.

Historically in Kubernetes, you assembled that experience yourself from a listener, an HTTPRoute that matched on the request body, an AgentgatewayBackend for each provider, and an AgentgatewayPolicy for the AI behavior. Every LLM deployment needed the same scaffolding, and that scaffolding stayed visible in your configuration.

The AgentgatewayModel API removes the scaffolding. Each resource declares one client-facing model and attaches directly to a Gateway listener. Agentgateway derives the routing, so you configure only what is specific to your setup.

Every model that attaches to the same listener is aggregated into a single model table. From that table, the listener serves the following behavior.

• Model extraction from the request body.
• The standard LLM API paths, such as /v1/chat/completions.
• Model discovery on /v1/models.
• Per-model provider routing.
• OpenAI-compatible error responses for unknown models.

Model-centric vs. route-centric configuration

Both approaches are supported. Use this table to choose.

Listener opt-in

A listener serves LLM traffic only when it explicitly allows the AgentgatewayModel route kind in allowedRoutes.kinds.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
      # Enables the listener's built-in LLM paths
      kinds:
      - group: agentgateway.dev
        kind: AgentgatewayModel

Adding the route kind turns on the listener’s built-in LLM paths. At first the listener has zero models, so every request returns a model_not_found error. Models become available as they attach.

A listener can allow both AgentgatewayModel and HTTPRoute, so LLM endpoints and ordinary HTTP routes coexist on the same port. When you set allowedRoutes.kinds, the listener accepts only the kinds that you list, so include every kind that you need.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
      kinds:
      - group: agentgateway.dev
        kind: AgentgatewayModel
      - group: gateway.networking.k8s.io
        kind: HTTPRoute

Concrete and virtual models

Every model is an AgentgatewayModel resource. A model either names a particular language model the provider that serves it (“concrete” model), or routes across other models (“virtual” model). The two fields are mutually exclusive.

The following diagram shows a client requesting the balanced virtual model, which splits traffic 80/20 across two internal concrete models. Each concrete model forwards to its provider.

    flowchart LR
  C[Client requests 'balanced'] --> V["Virtual model: balanced<br/>(weighted 80/20)"]
  V -->|80| F["Concrete model: fast<br/>Internal"]
  V -->|20| P["Concrete model: premium<br/>Internal"]
  F --> FP[Provider]
  P --> PP[Provider]
  style V fill:#7734be,color:#fff
  

Concrete models

A model that names its provider is a concrete model. It is an AgentgatewayModel that sets spec.provider, so the gateway knows which provider serves the request.

The following example shows the pieces that a concrete model configures. Only parentRefs and provider are required.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: gpt-5-mini
  namespace: agentgateway-system
spec:
  # The Gateway listener that serves this model
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  # The model name that clients request; defaults to metadata.name
  match:
    model: gpt-5-mini
  # Whether clients can request this model directly
  visibility: Public
  # The provider that serves this model
  provider: OpenAI
  # Optional: override the provider address
  baseURL: https://api.openai.com
  # Optional: policies that apply to this model only
  policies:
    auth:
      secretRef:
        name: openai-secret

Review the following table to understand this configuration.

Model matching

Use spec.match.model to control which model name in a request selects this resource. When you omit spec.match, the model matches metadata.name exactly.

For example, a model named gpt-5-mini with no spec.match serves requests that send "model": "gpt-5-mini" in the body. To publish a different name, set spec.match.model explicitly.

A match must use one of the following forms. Wildcards in any other position are rejected.

A wildcard model usually pairs with a model transformation, because the provider expects its own model name rather than the client-facing one. For an example, see Serve a model.

Model aliasing

A model resource is an alias by construction. The name that clients request is metadata.name or match.model, and the name that the provider receives comes from a model transformation. To publish fast as an alias for gpt-3.5-turbo, create a model named fast that rewrites the field.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: fast
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  provider: OpenAI
  policies:
    transformations:
    - field: model
      expression: '"gpt-3.5-turbo"'

Each alias is a separate resource, so it can carry its own credentials, authorization rules, and guardrails. For the AgentgatewayBackend equivalent, see Model aliasing.

Visibility

Use spec.visibility to control whether clients can request a model directly.

Set Internal on concrete models that exist only as virtual model targets.

Model policies

Concrete models accept an inline spec.policies block that supports the following policies. Each policy applies only to the model that declares it.

Virtual models cannot set spec.policies, because a virtual model has no provider of its own to authenticate to, transform for, or health check. Configure these policies on the concrete models that the virtual model targets.

Providers

Set spec.provider to one of the supported provider names, such as OpenAI. For a full list of providers, see the model provider API docs.

Some providers require a matching settings field.

• Azure requires spec.azure.
• VertexAI requires spec.vertexai.
• Bedrock requires spec.bedrock.
• Custom requires spec.custom.
• Ollama requires spec.baseURL.

Use spec.custom.backendRef to serve a model from a Kubernetes backend, such as an InferencePool.

Use spec.baseURL to override the provider address and base path prefix. It must be an absolute http or https URL with a host, and it cannot target localhost, loopback, or link-local addresses. Query parameters, fragments, and user info are not supported.

Virtual models

A model that routes across other models is a virtual model. It is an AgentgatewayModel that sets spec.virtualModel instead of spec.provider, so it has no provider of its own. It publishes one client-facing name and selects a concrete model to serve each request.

The following example splits traffic across two concrete models by weight.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: balanced
  namespace: agentgateway-system
spec:
  # The Gateway listener that serves this model
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  # Routing strategy across other AgentgatewayModel resources
  virtualModel:
    weighted:
      targets:
      - modelRef:
          name: internal-fast
        weight: 80
      - modelRef:
          name: internal-premium
        weight: 20

Routing strategies

Each virtual model uses exactly one strategy.

For examples of each strategy, see Virtual models.

Constraints

• Targets must be AgentgatewayModel resources in the same namespace. Cross-namespace references are not supported.
• Virtual models must be Public. The restriction stops virtual models from targeting each other, which could otherwise create routing loops.
• Virtual models cannot set spec.policies. Configure policies on the concrete target models instead.

Known limitations

• The API is experimental and turned off by default.
• An AgentgatewayPolicy cannot target an AgentgatewayModel.
• Status is not yet reported on the AgentgatewayModel resource. The status.parents field stays empty even when a model serves traffic correctly. To confirm that models attached, check the Gateway listener’s attachedRoutes count, or list the models on /v1/models.