agentgateway | Agent Connectivity Solved - Blog

MCP goes stateless with headers. Do you need an MCP-native data plane?

Tue, 21 Jul 2026 00:00:00 +0000

The next revision of the Model Context Protocol (MCP) is supposed to make implementation infrastructure (ie, gateways, routers, etc) easier to reuse/implement. The draft (2026-07-28) is intended to make every request self-contained and independently routable, and clean up anything that got in the way of that. Sessions, the initialize handshake, server-to-client callbacks, etc are all gone or deprecated.

If you expose MCP servers from a MCP/agentgateway, this is a consequential set of changes. And it’s already prompting a question I am hearing from platform teams: if MCP is now stateless and its routing fields live in HTTP headers, can’t I just use a plain L7 load balancer or API gateway?

The premise is misunderstood. The short answer is no! and the new spec actually makes a naive header-only proxy more dangerous, not less. This post walks through what changed, why a traditional proxy breaks under it, and what an MCP-native data plane like agentgateway does instead. We’ll even give you the most basic, runnable demo to prove it out.

What actually changed

Three “mega-SEPs” reshape everything else, and most of the rest of the draft falls out of them.

Stateless MCP (SEP-2575). The initialize / notifications/initialized handshake is removed. Every request now carries its own io.modelcontextprotocol/protocolVersion, clientInfo, and clientCapabilities in _meta, and a new mandatory server/discover RPC handles up-front capability and version negotiation. ping, logging/setLevel, and SSE resumability (Last-Event-ID) are gone.
Sessionless MCP (SEP-2567). The Mcp-Session-Id header and all session-lifecycle language are removed. List endpoints (tools/list, etc.) no longer vary per connection, which makes them cacheable. Cross-call state moves to explicit, server-minted handles passed as ordinary tool arguments.
HTTP transport standardization (SEP-2243). Routing fields are mirrored out of the JSON-RPC body into HTTP headers: Mcp-Method mirrors method, Mcp-Name mirrors params.name/params.uri, and servers can promote individual tool arguments into Mcp-Param-* headers via an x-mcp-header annotation in the tool schema.

Rounding it out: server-initiated requests (sampling, elicitation, roots callbacks) are replaced by the Multi Round-Trip Request pattern (SEP-2322); Tasks becomes an official extension (SEP-2663); and Roots, Sampling, and Logging are deprecated (SEP-2577).

The current finalized version (as of writing, one week before 7-28) is still 2025-11-25. agentgateway already implements the draft line as protocol version 2026-07-28, running the old and new protocols side by side, so we can talk about this concretely rather than hypothetically.

Wait!! “Now any load balancer can route MCP?”

Historically, routing information was “buried deep within the JSON-RPC payload,” forcing intermediaries to “terminate TLS and perform deep packet inspection to route traffic.” Lifting Mcp-Method and Mcp-Name into headers fixes that. And it genuinely does make coarse routing and rate-limiting easy for standard infrastructure.

And that’s exactly where it should stop. The mistake is extrapolating from routing to policy and trust. Once you decide the header is the source of truth, four things break: security, authorization UX, federation, and identity. Let’s dig into them.

Security: the header can lie to you

Headers and body are two copies of the same story, and nothing forces them to tell it the same way. Both are sent by the client. SEP-2243 anticipates exactly this and mandates that “any server processing the message body MUST validate that the HTTP header values exactly match the corresponding values in the JSON-RPC body.”

But a header-only proxy, by definition, never opens the body. Picture this: a support-desk agent is allowed to reach one MCP server through the gateway, and the platform team has locked it down to a single safe tool (echo) with a header allowlist. Reasonable enough. Except the same server also exposes printEnv. An attacker (or a prompt-injected agent) sends:

POST /mcp HTTP/1.1
Mcp-Method: tools/call
Mcp-Name: echo

{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"printEnv","arguments":{}}}

The proxy sees Mcp-Name: echo, decides “echo is safe,” and forwards. The header said echo. The body ran printEnv and dumped its environment variables, secrets and all. This is simple header/body mismatching and it’s been the root cause of a long line of CVEs in other protocols. Relying on the server SDK to catch it is not a great plan: it’s exactly the “someone else will validate it” assumption that these bugs live in.

An MCP-native gateway parses the body and validates the headers against it. Send that spoof through agentgateway and it never reaches the backend:

$ curl -sS http://localhost:3000/mcp \
 -H 'Content-Type: application/json' \
 -H 'Accept: application/json, text/event-stream' \
 -H 'Mcp-Method: tools/call' -H 'Mcp-Name: echo' \
 -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"printEnv","arguments":{}}}'

{"jsonrpc":"2.0","id":2,"error":{"code":-32020,"message":"Mcp-Name header/body mismatch"}}

(The full walkthrough of honest calls, method spoof, and policy denial is in Try it yourself at the end.)

The spoofed request is rejected with a JSON-RPC HeaderMismatch error (code -32020, the exact allocation SEP-2243 reserves) before it ever reaches a backend. The validation runs on every Streamable HTTP POST, ahead of any routing or session logic, and the check fires whenever a header is present and disagrees with the body, regardless of which protocol version the client negotiated.

Authorization and UX: filtering and graceful errors

Suppose you push authorization down to the header anyway. Two things go wrong immediately.

You can’t filter the tool list. tools/list is a body-level response; a header-only proxy can’t rewrite it. So a user authorized for 2 of 100 tools sees all 100, and the model repeatedly tries to call tools it can’t use. That’s a miserable agent experience, and it leaks the shape of your whole tool surface to every caller.

agentgateway evaluates policy against the real request and filters the list. Policies are CEL expressions that can reference the tool name, the call arguments, and JWT claims, not just the header-exposed name (examples/mcp-authorization):

mcp:
 policies:
 jwtAuth:
 issuer: agentgateway.dev
 audiences: [test.agentgateway.dev]
 jwks: { file: ./manifests/jwt/pub-key }
 mcpAuthorization:
 rules:
 - 'mcp.tool.name == "echo"' # anyone may echo
 - 'jwt.sub == "test-user" && mcp.tool.name == "get-sum"' # only test-user
 - 'mcp.tool.name == "get-env" && jwt.nested.key == "value"'

Tools that fail policy are dropped from tools/list, so the model only ever sees what the caller can actually use. This is fundamentally a body-and-identity operation; there is no header that carries it.

You return the wrong kind of error. In MCP, application-level failures are supposed to come back as JSON-RPC errors that the agent reads and routes around (“that tool isn’t available, I’ll try another path”). A naive proxy that answers a denied call with a bare HTTP 403 instead signals a transport- or auth-level failure, and clients treat that very differently, often tearing down the connection rather than gracefully continuing. agentgateway returns a structured JSON-RPC error the agent can act on, and it deliberately hides the tool’s existence rather than confirming it was denied:

{"jsonrpc":"2.0","id":4,"error":{"code":-32602,"message":"Unknown tool: printEnv"}}

That distinction matters more than it looks. “Access denied” confirms the tool is there and worth attacking; “unknown tool” hands a prober nothing to work with. The higher-level modes people are building on MCP (code mode, search mode, and similar) all depend on this body-level intelligence. None of it is reachable from headers.

Federation, and “free” statelessness

A big feature that nearly all of our users use: “Virtual MCP” where a single MCP endpoint exposed on the gateway exposes multiple backend MCP servers. Here is where header-only routing goes from limiting to impossible. Multiplexing (via the Virtual MCP concept) requires rewriting the body: namespacing tool names so they don’t collide, demultiplexing a tools/call back to the server that owns the tool, merging tools/list responses, and reconciling protocol versions across backends. agentgateway does all of this (examples/mcp-multiplex): tools are namespaced as target_tool, list operations fan out and merge, and calls are routed to the single owning backend. A header-only proxy has nowhere to put any of that logic.

The statelessness change does have a nice payoff worth calling. Under the old protocol, sessions pinned a client to a specific server instance; a proxy had to track and map session IDs to preserve that, and until proxies solved it you were effectively limited to a single replica per MCP server. agentgateway solved session mapping, but the draft makes the whole problem disappear for everyone: with no Mcp-Session-Id and self-contained requests, any request can go to any replica. Scaling an MCP backend horizontally becomes ordinary load balancing. That’s a real win, and credit goes to the MCP maintainers for designing the session problem out of existence rather than papering over it. It also frees the gateway to focus on trust, policy, and federation rather than session bookkeeping.

Identity: where the data plane carries the load

Most of the draft’s authorization work is about clients and servers, and much of it doesn’t strictly require an MCP-native proxy. But it leans heavily on there being a capable identity layer in the path, and that’s exactly the territory agentgateway has been building out. It’s already on main, with tests and examples.

OAuth resource server. agentgateway serves the .well-known/oauth-protected-resource and oauth-authorization-server metadata, issues WWW-Authenticate challenges, and validates JWTs across multiple issuers, with provider-specific handling for Auth0, Okta, Keycloak, and Descope.
Token exchange (RFC 8693). Incoming user tokens are exchanged for downstream-scoped tokens, with actor/subject token support and caching (examples/traffic-token-exchange).
Identity assertion / ID-JAG. A two-leg chained exchange (token-exchange → ID-JAG assertion → jwt-bearer → access token) for cross-application access, so an MCP call can carry a properly scoped, audience-bound identity to the backend (examples/traffic-cross-app-access).

This connects directly to the draft’s smaller auth items, including issuer (iss) validation for multi-AS mix-up protection (SEP-2468) and the multi-authorization-server migration guidance (SEP-2352), where per-issuer JWT validation is already in place.

The Key Takeaway

The draft’s thesis of self-contained, independently routable requests is the right direction, and it does make MCP easier to move through standard infrastructure. But moving bytes is not the same as enforcing trust. If anything, the new spec makes that boundary more important: a header-only proxy is now not just insufficient but actively unsafe.

If you’re working through stateless MCP behind a gateway and hitting these edges, I’d genuinely like to hear about it. Reach out on LinkedIn.

Try it yourself

The entire setup is one small config file. It puts agentgateway in front of a local MCP server (the reference server-everything), runs statelessly so every request is a self-contained POST, and adds a body-based policy that allows only the echo tool. Save it as agw.yaml:

mcp:
 port: 3000
 statefulMode: stateless  # each request is a self-contained POST; no session handshake
 targets:
 - name: everything
 stdio:
 cmd: npx
 args: ["-y", "@modelcontextprotocol/server-everything"]
 policies:
 cors:
 allowOrigins: ["*"]
 allowHeaders: ["*"]
 mcpAuthorization:
 rules:
 - 'mcp.tool.name == "echo"' # body-based authz: only `echo` is permitted

Start the gateway (needs npx for the stdio server):

agentgateway -f agw.yaml
# from a source checkout: cargo run -- -f agw.yaml

Now drive it by hand from another terminal. Every request needs Content-Type: application/json and Accept: application/json, text/event-stream.

1. Honest call: header matches body, tool is allowlisted → forwarded

curl -sS http://localhost:3000/mcp \
 -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
 -H 'Mcp-Method: tools/call' -H 'Mcp-Name: echo' \
 -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"echo","arguments":{"message":"hello"}}}'
# data: {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"Echo: hello"}]}}

2. The attack: header says echo, body calls printEnv (env exfiltration) → rejected

curl -sS http://localhost:3000/mcp \
 -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
 -H 'Mcp-Method: tools/call' -H 'Mcp-Name: echo' \
 -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"printEnv","arguments":{}}}'
# {"jsonrpc":"2.0","id":2,"error":{"code":-32020,"message":"Mcp-Name header/body mismatch"}}

3. Method spoof: header tools/list, body tools/call → rejected

curl -sS http://localhost:3000/mcp \
 -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
 -H 'Mcp-Method: tools/list' -H 'Mcp-Name: echo' \
 -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"echo","arguments":{"message":"x"}}}'
# {"jsonrpc":"2.0","id":3,"error":{"code":-32020,"message":"Mcp-Method header/body mismatch"}}

4. Honest but denied by the body policy: printEnv blocked, tool hidden

Headers are consistent, but the policy allows only echo. Note the response says Unknown tool rather than “access denied”: the tool’s existence is hidden, not just its authorization.

curl -sS http://localhost:3000/mcp \
 -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
 -H 'Mcp-Method: tools/call' -H 'Mcp-Name: printEnv' \
 -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"printEnv","arguments":{}}}'
# {"jsonrpc":"2.0","id":4,"error":{"code":-32602,"message":"Unknown tool: printEnv"}}

Explore the docs and get started today.
Star and contribute on GitHub.
Join the conversation on Discord.
Attend our weekly community meetings.

Reduce LLM Spend with Semantic Routing

Mon, 20 Jul 2026 00:00:00 +0000

I spend a lot of time in Go and Rust, where a quick “one more thing” in a long coding chat can turn a tidy refactor into an hour of debugging. After the hard part is over, routine tests and follow-up questions can still burn premium-model credits. Sending every request to a cheaper model is not the answer: distributed-systems design, correctness work, and difficult debugging need more capable help.

This example combines vLLM Semantic Router (vSR) with agentgateway to send routine Go and Rust coding tasks to gpt-5.4-nano and escalate complex correctness, distributed-systems, and deep-debugging work to gpt-5.5. The goal is a reproducible cost reduction without quietly routing everything to the cheapest tier.

Make the Decision Observable

One model name in, the appropriate model tier out - with the routing decision and realized cost visible at the gateway

The client sends an OpenAI-compatible request with model: "auto". An agentgateway policy sends it to vSR, which evaluates semantic, complexity, keyword, context, and structure signals before selecting a model. Agentgateway forwards the request and records the outcome. Cost is not a classifier input; it is measured after the request, keeping the policy explainable.

auto opts into that automatic selection. A client can instead request gpt-5.4-nano or gpt-5.5 directly to force a tier and bypass automatic routing; the request still passes through agentgateway. Organizations that require automatic routing can use a request-body transformation to rewrite every request to auto.

Price the Outcome at the Gateway

My company needs prices, not just token counts, to make a budget decision. Agentgateway loads a model cost catalog and exposes realized request cost in metrics, logs, and traces. The model cost catalog guide explains how to generate and load the catalog with agctl.

For this evaluation, I use the integrated OpenTelemetry stack. Prometheus, Loki, Tempo, the OpenTelemetry Collector, and Grafana let engineering and finance inspect the same model-selection event.

A Small, Reproducible Evaluation

I built the cost-based semantic routing demo around a compact 50-prompt Go and Rust dataset. Half of the prompts are routine implementation or test work; the other half involve correctness or distributed-systems problems. I send every prompt through two lanes:

Lane	Purpose
`routed`	vSR selects `gpt-5.4-nano` or `gpt-5.5`.
`always_expensive`	Every request uses `gpt-5.5`; the cost baseline.

Each run has an ID on every request, so its cost query excludes unrelated cluster traffic. I deliberately leave out a forced-cheap lane: the useful comparison is whether routing saves money while still escalating complex work.

Each prompt has an expected tier in the checked-in dataset: 25 routine prompts expect gpt-5.4-nano, and 25 complex prompts expect gpt-5.5. Expected-tier agreement is the share of requests where vSR selected that labelled tier. It checks the routing policy, not the quality of the model’s answer.

In a clean run on July 16, 2026, all 100 primary requests completed with exact model-catalog lookups. vSR selected gpt-5.4-nano for 30 requests and gpt-5.5 for 20. Catalog-priced agentgateway metrics reported $0.591444 for the routed lane versus $0.914840 for the always-expensive lane, a 35.4% cost reduction. Routed p50 latency was 5.22 seconds, compared with 7.55 seconds for the always-expensive lane. P95 was 19.09 seconds for routed traffic and 19.61 seconds for the always-expensive lane.

The chart reached 90% expected-tier agreement (45 of 50 prompts). All 25 routine prompts stayed on nano, and 20 of 25 complex prompts escalated to GPT-5.5. This is not a claim of perfect model selection, but it shows the savings did not come from a blanket downgrade: 40% of routed requests still used the expensive model. It is a policy sanity check, not an answer-quality benchmark or a substitute for application task-success and user-feedback data.

The default dataset uses independent requests. For long-running coding agents, the optional cache-transition evaluation uses two ordered ten-turn Go and Rust conversations with a stable prefix. It reports provider-observed cache reads by model transition without mixing cache behavior into the primary evaluation.

Try It

The demo automates the setup work I would otherwise repeat by hand. It uses the agentgateway semantic-routing example, creates or reuses a kind cluster, and installs agentgateway, vSR, a model cost catalog, and the OpenTelemetry stack. It checks readiness, routing, catalog-priced metrics, logs, and traces before the primary requests run.

Before running it, review the demo requirements: 12 GiB of Docker memory, 30 GiB of free disk, the required CLIs, an OpenAI API key, and access to both models.

git clone https://github.com/danehans/agentgateway-demos.git
cd agentgateway-demos/cost-based-semantic-routing

export OPENAI_API_KEY='sk-...'
./demo.sh all --yes

The default run sends 106 billable requests: two routing probes, four smoke-test requests, and 100 primary requests. It writes request-level JSONL, an evaluation-scoped Prometheus report, and an SVG chart under results/.

When I want to change the policy, I edit the fetched vSR values, redeploy with ./demo.sh router, and run ./demo.sh eval --yes again.

Use It From Codex

Codex lets me choose a model manually, but a user-level profile can instead send every request to a gateway such as agentgateway with the stable auto model name:

# ~/.codex/agentgateway.config.toml
model = "auto"
model_provider = "agentgateway"

[model_providers.agentgateway]
name = "Corporate agentgateway"
base_url = "https://my.corp.agentgateway.com/v1"
wire_api = "responses"

I start Codex with codex --profile agentgateway. It sends OpenAI Responses API traffic to agentgateway; vSR selects the tier and agentgateway records the decision and realized cost.

Current Codex emits an unknown model auto fallback-metadata warning. Routing still works, but Codex warns that the fallback metadata may degrade behavior.

The profile keeps credentials out of the configuration file. Codex documents custom model providers and user-level configuration profiles.

From Example to Rollout

Semantic routing is not a magic cost switch. Start with an explainable policy, compare it with always using the expensive model, confirm it uses both tiers, and tune it against task completion, feedback, retries, and escalation rates. Agentgateway supplies the accounting and observability; vSR makes the semantic decision.

This example is intentionally static: it uses configured keywords, semantic candidates, weights, and thresholds rather than learning from application outcomes. Those signals need periodic review as traffic changes. The agentgateway community and vSR project are continuing work to make semantic routing easier to tune, operate, and measure against real application traffic, so stay tuned.

What’s Next

After validating a tiering policy, use agentgateway cost controls to attribute usage with virtual keys, observe realized spend by model and consumer, and enforce budget or spend limits.

How One Platform Team Scaled AI Adoption with Kagent and Agentgateway

Mon, 13 Jul 2026 00:00:00 +0000

For many enterprise platform teams, the challenge isn’t a lack of ideas for AI. It’s finding a way to scale support and automation without adding headcount, compromising security, or creating another platform to maintain.

That was the challenge facing the platform team who run payments for DNB, Norway’s largest financial services group.

After attending KubeCon Europe in Amsterdam (which David Karlsen jokingly refers to as “AICon” because of the sheer amount of AI content), the team returned with a simple question: how could they use these new ideas to make themselves more effective?

What started as an idea for a Slack bot evolved into a broader agentic platform. Using kagent and agentgateway, the team built a foundation that could connect Slack, internal systems, and MCP servers while keeping security and access control front and center.

From “How Do We Scale Ourselves?” to a Working System

The starting point wasn’t AI experimentation. It was a practical operational challenge.

“We came back home and thought, how can we kind of automate our team so we are more efficient with the same set of hands?” — David Karlsen, Staff Engineer at DNB

Like many platform teams, they were responsible for infrastructure, architecture, and internal support. As demand grew, they needed a way to help more users without increasing operational overhead.

Their first instinct was to build a Slack bot. But the team quickly realized they wanted something broader than a chat interface. They wanted an architecture that could support multiple user experiences while keeping intelligence, access control, and integrations cleanly separated.

That decision became foundational to the project.

Why They Didn’t Just Build a Slack Bot

Rather than building a single tightly coupled assistant, the team adopted a modular architecture using agentgateway and kagent.

As David put it:

“We don’t want our bot to be too smart. We want to build the smartness into the agents.”

That idea became one of the key architectural decisions behind the project.

At a high level, the design looked like this:

Users (Slack, IDEs) → Agentgateway → Kagent-managed Agents → Agentgateway → MCP Servers → Enterprise Systems

In this architecture:

User interfaces handle interaction.
Agents handle reasoning and orchestration.
MCP servers expose tools and systems.
Agentgateway controls access and policy enforcement.
LLM access is gated by agentgateway’s virtual-LLM pattern, for flexible model access, tracing, and ease of access to LLMs.
OTEL-based traces to track calls, outcomes, and actors.
Langfuse to track LLM traffic, tool calls, and support learning and optimization.

This separation gave the team something more valuable than a chatbot: a reusable control plane for agentic experiences where users may have different experience and background:

Chat with agents in kagent.
Chat with agents via Claude Code via kagent agents exposed as an MCP server behind agentgateway.
Chat directly with their MCP servers.

Different users could access the same underlying capabilities through different interfaces while security, governance, and observability remained consistent.

Why Kagent and Agentgateway Fit Their Environment

The team evaluated several approaches before settling on kagent and agentgateway.

One alternative would have required a more intrusive installation model and a new networking stack that didn’t fit their environment. Other solutions required additional configuration and operational complexity.

Kagent stood out because it was straightforward to deploy and easy to extend.

“After we tried kagent we said, okay, this is pretty easy to use, easy to add agents, easy to add MCP servers. We like it that everything is configurable. You have to just provide a manifest and everything got deployed nicely.” — Ibrahim Umar, Senior DevOps Engineer at DNB

That declarative model aligned naturally with the team’s existing Kubernetes, OpenShift, and GitOps workflows.

The ability to deploy agents and MCP servers through declarative YAML meant they could experiment quickly without introducing operational friction.

Just as importantly, agentgateway remained a separate control layer rather than being bundled into a monolithic platform with kagent. That flexibility allowed the team to manage their MCP servers’ access independently from agent deployment and orchestration.

Security and Identity from Day One

Because the team operates in an enterprise environment and works with real systems, security requirements shaped the architecture from the beginning.

Agentgateway was integrated with Microsoft Entra ID and configured to use JWT-based identity propagation and authorization controls. This allowed the team to enforce identity-aware policies while remaining aligned with existing Azure-native authentication workflows.

As the project evolved, they also explored broader questions around agent identity, delegated access, and authorization boundaries.

“Security needs to be in the driver’s seat.” — David Karlsen, Staff Engineer at DNB

The team viewed security as a prerequisite for adoption rather than something to be added later. That perspective influenced everything from gateway design to access control decisions.

Real Use Cases: Debugging, Introspection, and Operational Support

One of the most important aspects of the project was deciding what agents should do—and what they shouldn’t do.

The team deliberately focused on use cases where AI could accelerate understanding and troubleshooting without introducing unnecessary operational risk.

One of the first MCP servers Ibrahim built connected to DB2, a system many enterprises still depend on. Within a short time, the team could inspect database structures, review lock reports, and look at performance data through natural language interactions instead of manually digging through tools and dashboards.

The same pattern extended to other systems as well. The team connected MQ, Jira, Atlassian tools, and other internal platforms to create a broader developer support ecosystem capable of helping users navigate increasingly complex environments without understanding their specific APIs.

Adoption Happened Faster Than Expected

Many AI initiatives struggle not because of technology limitations but because users don’t change their habits.

That wasn’t the case here.

The team had already invested heavily in Slack and digital-first collaboration. Because the assistant appeared in a familiar environment, users could begin interacting with it immediately without learning an entirely new workflow.

The result was faster adoption than the team anticipated.

The team didn’t spend weeks training users. People simply started talking to the bot in Slack. Because the organization had already invested heavily in digital-first collaboration, adding another helper into the same environment felt natural.

One night, David found himself asking the assistant about a problem in the assistant itself. Because the agent had access to its own source code and traces, it was able to explain what was happening and help diagnose the issue.

“It was a bit meta but fascinating,” David said. “It could actually explain very well what was going on.”

It wasn’t a planned use case, but it was one of those moments that made the team’s investment in the architecture feel worthwhile.

Why Open Source and Open Standards Mattered

For the team, open source wasn’t simply a philosophical preference. It was a practical requirement.

Both David and Ibrahim have long histories with open source software, so transparency mattered. When they hit a roadblock, they wanted to inspect the code, understand the architecture, and potentially fix the problem themselves rather than wait for someone else to solve it. And they wanted the freedom to adapt the architecture as requirements evolved.

“Being an open source project is very fascinating… if we have time, we try to implement it ourselves and contribute.” — Ibrahim Umar, Senior DevOps Engineer at DNB

The team also valued support for open standards including:

Model Context Protocol (MCP)
Agent-to-Agent (A2A) communication
JWT-based identity propagation

These standards helped reduce lock-in while making it easier to connect existing systems to the emerging agentic ecosystem.

In many cases, they didn’t need to rebuild services at all. Existing REST APIs could be exposed as MCP servers easily via agentgateway, creating a practical bridge between traditional enterprise systems and modern AI workflows.

That approach allowed them to build on investments they had already made rather than introducing an entirely new platform from scratch.

From Users to Contributors

As the team became more familiar with the architecture, they moved beyond adoption and started contributing ideas back to the kagent open source community.

Their path into the community started with a real problem.

While evaluating the architecture, David began digging into security boundaries and access enforcement. That investigation led him into the project’s backlog, Discord discussions, and eventually to writing a design proposal for a security enhancement. They look forward to working with the kagent community to implement this feature.

What They Learned

Looking back, several lessons stood out:

Security and authorization need to be designed up front.
A clean separation between bots, agents, gateways, and MCP servers makes systems easier to scale and reuse.
The declarative agent deployment model provided by kagent fits naturally with Kubernetes and GitOps operating models.
Open standards reduce lock-in and make ecosystems easier to extend.
Existing REST APIs can often be bridged into MCP using agentgateway rather than rebuilt.
Flexibility matters because identity, context management, and agent interoperability continue to evolve rapidly.

For teams evaluating agentic AI, their recommendation is simple: start with a concrete use case, understand the security model thoroughly, and build on composable foundations that can evolve over time.

Final Thoughts

When the team came back from KubeCon, they weren’t looking for a new AI project. They were trying to answer a practical question:

How do we become more efficient with the same set of hands?

For them, the answer wasn’t a smarter chatbot. It was a modular architecture built around agents, MCPs, and clear security boundaries. Kagent and agentgateway provided the building blocks, but the bigger lesson was that successful AI adoption starts with solving a real operational problem, not with chasing the latest trend.

Agentgateway adds token exchange, jwt-assertion, and Entra OBO

Sun, 12 Jul 2026 00:00:00 +0000

Shielding AI agents from sensitive MCP / API credentials (secrets, API keys, tokens, etc) is the prevailing best practice pattern to keep sensitive secrets from leaking into AI model conversations or across AI agent boundaries. OpenClaw, for example, has direct file system access in its most general deployment and can read files/secrets/env variables, etc and could easily send these secrets in any requests.

The right way in an enterprise environment: user identity + agent identity tied to an authorization grant, which determines what is allowed to be done. Any calls to MCP servers or APIs have their credentials injected transparently by the infrastructure. The cornerstone of this injection is exchanging an agent or user’s identity for the correct secrets/API keys/tokens.

In the upcoming agentgateway release, we have opensourced two new exchange capabilities, including a bonus third exchange opportunity with Microsoft Entra — all under backendAuth.oauthTokenExchange:

Grant	Spec	Subject sent as	Typical IdP
Token exchange	RFC 8693	`subject_token`	Keycloak, Okta, ZITADEL, Auth0, …
JWT bearer (jwt-assertion)	RFC 7523	`assertion`	Keycloak JWT Authorization Grant, many AS
Entra on-behalf-of (OBO)	Entra’s OBO (jwt-bearer + extras)	`assertion`	Microsoft Entra ID

Entra is the important enterprise wrinkle: it does not speak RFC 8693. Its OBO flow is jwt-bearer with requested_token_use=on_behalf_of. Same outcome (user-scoped downstream token), different grant shape — so both grants live under one oauthTokenExchange block.

RFC 8693 token exchange

Default grant. The gateway POSTs the inbound bearer as subject_token to your IdP’s token endpoint and attaches the returned access token upstream.

backendAuth:
 oauthTokenExchange:
 host: idp.example.com:443
 tokenEndpointPath: /oauth2/default/v1/token
 clientAuth:
 clientId: gateway-client
 clientSecret: $OAUTH_CLIENT_SECRET
 method: clientSecretBasic
 audiences:
 - upstream-api
 scopes:
 - read

What the gateway sends (conceptually):

grant_type=urn:ietf:params:oauth:grant-type:token-exchange
subject_token=<inbound bearer>
subject_token_type=urn:ietf:params:oauth:token-type:access_token
audience=upstream-api
scope=read

Optional knobs that matter in agentic setups:

actorToken — RFC 8693 delegation (act claim). Useful when an agent identity is acting for a user.
resources — RFC 8707 resource indicators when the AS expects them (common for MCP / API audiences).
subjectToken.source — read the subject from a non-default header, cookie, or CEL expression instead of Authorization.
cache — in-memory cache keyed by subject + grant params; TTL capped by the subject JWT exp.

Runnable example: examples/traffic-token-exchange/oauth-rfc8693.

JWT assertion (RFC 7523 jwt-bearer)

Flip grantType: jwtBearer and the same policy speaks JWT bearer instead of token exchange. The inbound credential becomes the assertion (not subject_token). requestedTokenType / actorToken are rejected for this grant — they belong to RFC 8693.

backendAuth:
 oauthTokenExchange:
 host: idp.example.com:443
 tokenEndpointPath: /realms/backend-oauth/protocol/openid-connect/token
 grantType: jwtBearer
 clientAuth:
 clientId: requester-client
 clientSecret: requester-secret
 method: clientSecretBasic
 audiences:
 - target-client

Form shape:

grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
assertion=<inbound token>
audience=target-client

This grant works for any authorization servers that support JWT Assertion Grant (RFC 7523). This is the grant Keycloak’s JWT Authorization Grant (preview in 26.5+) implements across realms — present a JWT from realm A, get a token from realm B that trusts A. Full two-realm walkthrough: examples/traffic-token-exchange/jwt-authz-grant.

Microsoft Entra OBO

Entra OBO is jwt-bearer with provider-specific extras. The hand-written request looks like:

curl -X POST "https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "client_id=<CLIENT_ID>" \
 -d "client_secret=<CLIENT_SECRET>" \
 -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer" \
 -d "requested_token_use=on_behalf_of" \
 -d "scope=https://graph.microsoft.com/.default" \
 -d "assertion=<USER_ACCESS_TOKEN>"

Agentgateway produces that same request from config:

backendAuth:
 oauthTokenExchange:
 host: login.microsoftonline.com:443
 tokenEndpointPath: /<TENANT_ID>/oauth2/v2.0/token
 grantType: jwtBearer
 clientAuth:
 clientId: <CLIENT_ID>
 clientSecret: <CLIENT_SECRET>
 method: clientSecretPost  # credentials in the BODY (Entra expects this shape)
 scopes:
 - https://graph.microsoft.com/.default
 additionalParams:
 requested_token_use: '"on_behalf_of"' # CEL string literal

Microsoft form field	Produced by
`grant_type=...jwt-bearer`	`grantType: jwtBearer`
`assertion=<user token>`	inbound bearer (jwt-bearer sends subject as `assertion`)
`scope=...`	`scopes:` (joined into one space-delimited `scope`)
`client_id` + `client_secret` in body	`clientAuth.method: clientSecretPost`
`requested_token_use=on_behalf_of`	`additionalParams` (CEL)

Gotchas worth calling out:

requested_token_use is a vendor extension, so it lives in additionalParams, not a first-class field. Values are CEL expressions — a literal string needs the inner quotes: '"on_behalf_of"'.
Prefer clientSecretPost for Entra so credentials land in the form body (matching Microsoft’s docs). Default clientSecretBasic would put them in an Authorization: Basic header instead.
Inbound JWT validation still belongs on the route (jwtAuth / MCP auth). Exchange only runs after the gateway has accepted the user.

This is the missing piece after enterprise MCP SSO with Entra: SSO gets the user into the gateway; OBO gets the user out to Graph (or any Entra-protected API / MCP server) without re-prompting login.

Kubernetes: same knobs as a policy

On Kubernetes the same exchange attaches via AgentgatewayPolicy on a Service or Backend:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: okta-token-exchange
spec:
 targetRefs:
 - group: ""
 kind: Service
 name: my-backend
 backend:
 auth:
 oauthTokenExchange:
 tokenEndpoint:
 group: agentgateway.dev
 kind: AgentgatewayBackend
 name: okta-token-endpoint
 port: 443
 path: /oauth2/default/v1/token
 clientAuth:
 clientId: "<okta-client-id>"
 secretRef:
 name: okta-oauth-client
 scopes:
 - "<scope>"

Token endpoints are backend references (not raw URLs), so TLS, DNS, and ReferenceGrant rules stay consistent with the rest of the data plane. Client secrets come from Kubernetes Secrets; privateKeyJwt is available when your AS (Okta, etc.) requires signed client assertions.

Beyond single-leg exchange

If every MCP tool call just reused the inbound enterprise JWT, every upstream would see a token that was never minted for it. If every agent SDK keeps its own token vault, you get N credential stores and zero consistent audit which is exactly the OpenClaw-style leak surface from the earlier in this post. Gateway-side exchange keeps secrets out of the agent, preserves user identity (and optional actorToken for agent-on-behalf-of-user), and caches so agent loops don’t hammer the token endpoint.

These single-leg grants are also the building blocks for Cross App Access / ID-JAG (a two-leg composition of token exchange and JWT-bearer). See the traffic-cross-app-access examples if you want to explore that path. Will cover that in depth in future blogs.

Try it

Standalone examples: RFC 8693 · JWT bearer + Entra OBO shape
Backend auth docs: standalone backend authentication
Related reading: Enterprise MCP SSO with Entra · PR #2189 (data plane) · PR #2458 (Kubernetes controller)

Explore the docs and get started today.
Star and contribute on GitHub.
Join the conversation on Discord.
Attend our weekly community meetings.

MCP Guardrails with AARM and Microsoft Agent Governance Toolkit

Mon, 29 Jun 2026 00:00:00 +0000

Agentgateway can call out to policy engines for LLM guardrail or for enterprise policy decisions. We’ve recently added support for MCP guardrails. This blog goes a layer deeper and shows where MCP guardrails would fit into an agentic architecture following the principals from the AARM paper.

TL;DR

I have put together a demo that digs into the AGT framework with agentgateway.

You can find it here: https://github.com/christian-posta/agent-governance-agw

AI agents are not like microservices. An agent’s intent is “interpreted” and explores its way to a solution for a goal. When agentgateway sits on the request path to other agents, MCP tools, or APIs as a policy enforcement point (PEP), we are one step closer to coralling this emergent behavior with policy and guardrails.

Typical policy decision systems (think things like OPA) are good at evaluating a number of signals and making policy decisions. Think “attribute based access control”. They are stateless, take some “context” to evaluate (subject, roles, the resource being accessed, the action being taken) and return a deterministic allow/deny decision.

For traditional API calls, this works brilliantly. The context is predictable and discrete: user A with role B is trying to invoke operation C on resource D. Write the Rego rule, ship the policy, done.

But AI agents break this model. The same MCP tool call can be entirely legitimate in one context and a serious security or compliance violation in another, and the difference isn’t captured in any single call’s attributes. We have to dig into the pattern of calls and the intent behind them. An agent that reads a config file, then queries a database for credentials, then calls an external HTTP endpoint hasn’t violated any individual policy rule in isolation. Each step may be fully permitted. The threat is the sequence.

Autonomous Agent Runtime Management (AARM)

This is where agent-native governance toolkits like Microsoft’s Agent Governance Toolkit (AGT) come in. AGT is based on the excellent “AARM” paper (Autonomous Action Runtime Management) from Hermano Errico.

So what is it that AARM / AGT add on top of something like OPA? Actually these systems do use OPA/cedar/whatever under the covers. So specifically from what I can see it adds is:

Context / tool call accumulation (history of previous calls, ideally hash-chained)
Context dependent policy over and above allow/deny
Intent drift - evaluate API/tool calls and next step tool calls in terms of original intent
Signed receipts about decisions made given action + context

All of this can be fed into the stateless policy engine to get richer decisioning. I recommend reading the AARM paper for more.

Beyond richer context, AARM also introduces a richer decision vocabulary than a plain allow/deny. The paper defines four action categories that the policy engine must be able to return:

Decision	Meaning
Forbidden	Hard block regardless of context. No session history needed — these are absolute organizational limits (e.g., `rm -rf /`, known-malicious endpoints). Static policy suffices.
Context-dependent deny	The action is policy-permitted in isolation but blocked because accumulated session context reveals inconsistency with the user’s original intent. Classic example: reading customer PII then immediately emailing an external address — neither action alone triggers a violation, but the composition does.
Context-dependent allow	Denied by default but permitted when context demonstrates clear alignment with legitimate intent. Deleting database records looks dangerous in isolation; if the session confirms the user said “clean up my test data,” blocking it is wrong. Context transforms a default-deny into an informed allow.
Context-dependent defer / escalate	Risk cannot be conclusively determined from available context. Rather than committing to an unsafe allow or deny, execution is suspended and escalated for human approval — for example, a credential rotation outside a maintenance window where the context is ambiguous.

The AGT implementation maps these to verdict decisions of allow, deny, warn, escalate, and transform. The transform verdict is particularly interesting: instead of rejecting a response outright, AGT can mutate it by redacting PII from a tool result before it reaches the agent’s context window, for instance.

One of the big things that stood out to me in the AARM approach is the need for a trusted component to do things like session accumulation and runtime enforcement of policy. It can be done within the agent (SDK), but that’s technically within an untrusted domain. That’s where agentgateway fits into the picture.

Microsoft Agent Governance Toolkit with Agentgateway

Microsoft’s Agent Governance Toolkit is an implementation of the AARM paper. AARM specifies two main ways to implement its ideas: SDK or proxy/gateway. AGT provides for both. We are going to look at the proxy approach.

Before looking at the configuration, it helps to understand where in the agent lifecycle AGT can intervene. The ACS (Agent Control Specification) — the policy layer inside AGT — defines eight intervention points that span the full agent loop:

Intervention point	When it fires
`agent_startup`	Before the agent run begins — evaluate session metadata and identity
`input`	At request ingress, before the agent loop starts
`pre_model_call`	Before the LLM is called — inspect messages, context, and tool definitions
`post_model_call`	After the model responds, before the host acts on it
`pre_tool_call`	Before each tool/MCP call executes
`post_tool_call`	After each tool result, before it returns to the agent
`output`	On the assembled final response to the user
`agent_shutdown`	On session termination — evaluate summaries and audit metadata

This matters for the agentgateway integration because agentgateway covers several of these intervention points directly at the network layer: the mcpGuardrails feature maps onto pre_tool_call and post_tool_call, while agentgateway’s existing LLM guardrails cover pre_model_call and post_model_call. The remaining points (startup, input, output, shutdown) can be covered by the AGT SDK inside the agent itself, giving you defense in depth across both the application layer and the network layer.

Agentgateway 1.3 recently added a new “mcpGuardrails” functionality (delivered in PR #1842) to complement its existing LLM guardrails capabilities and External Authz callouts.

The wire protocol is modeled directly on Envoy’s ext_authz, but operates at the JSON-RPC method layer of MCP — gating and mutating individual methods like tools/call, tools/list, prompts/get, and resources/read — rather than at the raw HTTP layer. A remote gRPC policy server implements a two-method service:

service ExtMcp {
 rpc CheckRequest(McpRequest) returns (McpRequestResult);
 rpc CheckResponse(McpResponse) returns (McpResponseResult);
}

That split matters: ext_authz only sees the inbound request, but mcpGuardrails also gives the policy server a clean shot at the response after agentgateway has merged any fanned-out results into the client-facing view. This is what makes transform-style verdicts (redacting PII from a tool result, for example) feasible without each policy server having to re-implement MCP framing and multiplexing.

For example, we can configure this mcpGuardrails like this:

 mcp:
 guardrails:
 processors: # ordered chain, first reject short-circuits 
 - methods: # allowlist with phase per method
 "tools/call": Full  # Request | Response | Full | Off
 "*/list": Response  # exact, `prefix/*`, `*/suffix`, or `*`
 remote:
 backendRef: # Service or Backend 
 name: my-policy-server
 failureMode: FailClosed  # or FailOpen 
 metadata: # CEL → google.protobuf.Struct
 tenant: "request.headers['x-tenant']"
 allowedRequestHeaders: [x-tenant] 
 disallowedRequestHeaders: [":authority"]

With the mcpGuardrails config we can:

Fire per-method checks. Each processor declares an allowlist of MCP methods it cares about, keyed by method name. Keys can be exact (tools/call), prefix wildcards (tools/*), suffix wildcards (*/list), or * for everything.
Choose the phase per method. Each entry’s value is one of Request, Response, Full, or Off. `
Get a single check for virtual-MCP fanouts. When agentgateway multiplexes several upstream MCP servers behind one logical endpoint, methods like tools/list fan out to every backend. Even so, agentgateway fires one CheckRequest for the whole client call (with service_names listing all targets) and one CheckResponse on the merged result, so policy servers see exactly what the client sees, not N partial views.
Return Pass / Reject / Mutated. A Mutated request replaces the JSON-RPC params bytes before they reach the upstream; a Mutated response replaces the JSON-RPC result before the client sees it. This is what enables AGT’s transform verdict — redact a field from a tools/call argument, drop a tool from a tools/list result, scrub a PII column from a database query response.
Pick a failure mode. failureMode: FailClosed (default) rejects the call if the policy server is unreachable or returns garbage; FailOpen lets it through. The same setting governs gRPC errors and protocol violations (e.g. a Mutated response that doesn’t parse back into a valid ServerResult).
Compose processors. The processors list is an ordered chain; the first to Reject short-circuits, mutations from earlier processors are visible to later ones, and metadata maps merge across the chain.

I have put together a demo that digs into the AGT framework with agentgateway.

You can find it here: https://github.com/christian-posta/agent-governance-agw

Try it and get involved

Ready to put MCP guardrails in front of your own agents and tools? Start with the standalone or Kubernetes quickstart, then follow the MCP guardrails docs to wire up an external policy server. Clone the AGT + agentgateway demo to walk through the full AARM flow end to end.

Explore the docs and get started today.
Star and contribute on GitHub.
Join the conversation on Discord.
Attend our weekly community meetings.

Benchmarking Agentgateway vs LiteLLM Part 2: Fixed Throughput

Fri, 26 Jun 2026 01:00:00 +0000

In Part 1, I pushed both agentgateway and LiteLLM to their maximum throughput and compared latency, CPU, and memory usage. The downside of that benchmark was that each gateway was processing a very different workload. agentgateway handled roughly 10× the requests of LiteLLM so it wasn’t exactly an apples-to-apples comparison.

In this post, I use a fixed target throughput of 3,000 QPS for both gateways. This allows me to compare how efficiently each proxy handles the same traffic level.

Test setup

The benchmark uses a very simple architecture. A mock LLM server immediately returns a fixed response so the benchmark measures proxy overhead rather than model inference time.

Fortio generates traffic against each gateway at the configured rate.

fortio (bt) ──► litellm :4000 ───────┐
├──► mock-server (hyper-server) :8081
fortio (bt) ──► agentgateway :4001 ──┘

I ran the benchmark with:

./scripts/run-benchmark.sh -q 3000 -d 30

The benchmark uses:

32 concurrent connections
Target throughput: 3,000 QPS
1 KB request payloads
30-second benchmark duration

Results

Throughput & Latency

Gateway	Actual Throughput	P50	P90	P99
agentgateway	2998.94 QPS	0.227 ms	0.249 ms	0.436 ms
LiteLLM	2465.89 QPS	12.318 ms	19.739 ms	30.626 ms

A few observations immediately stand out.

First, agentgateway sustained almost exactly the requested throughput of 3,000 QPS, processing 89,984 successful requests over the 30-second benchmark.

LiteLLM never reached the target rate. It averaged 2,466 QPS, about 18% below the requested throughput, completing 74,008 requests during the same period.

Latency was also dramatically different. agentgateway maintained a P99 latency below half a millisecond, while LiteLLM’s P99 exceeded 30 ms. Even median latency (P50) was over 50× lower with agentgateway.

CPU & Memory

Gateway	Avg CPU	Peak CPU	Avg Memory	Peak Memory
agentgateway	13.4%	29.5%	13 MiB	17 MiB
LiteLLM	345.5%	1158.5%	11.67 GiB	11.69 GiB

The resource utilization is arguably even more interesting than the latency numbers.

At essentially the same request rate, agentgateway used only 13% average CPU, while LiteLLM averaged 345% CPU, roughly 26× higher.

Memory usage showed an even larger gap. agentgateway stayed around 13 MB throughout the test, whereas LiteLLM consumed nearly 12 GB of RAM.

This means agentgateway handled a higher request rate while using only a tiny fraction of the system resources.

Raw benchmark output

./scripts/run-benchmark.sh -q 3000 -d 30
==> Run ID: 20260626-165414
==> LiteLLM workers: 18

Running fortio to litellm at 3000 QPS for 30s and 32 connections...
qps: 2465.89qps p50: 12.318ms p90: 19.739ms p99: 30.626ms

Running fortio to agentgateway at 3000 QPS for 30s and 32 connections...
qps: 2998.94qps p50: 0.227ms p90: 0.249ms p99: 0.436ms

DEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99
litellm,fortio,3000,32,30,1104,74008,2465.89qps,12.318ms,19.739ms,30.626ms
agentgateway,fortio,3000,32,30,1104,89984,2998.94qps,0.227ms,0.249ms,0.436ms

CONTAINER,SAMPLES,AVG_CPU%,PEAK_CPU%,AVG_MEM,PEAK_MEM
perf-agentgateway,21,13.42%,29.47%,13.15MiB,17.07MiB
perf-litellm,21,345.51%,1158.47%,11.67GiB,11.69GiB
perf-mock-server,21,6.23%,8.59%,3.06MiB,3.17MiB

Visualized results

I asked Cursor to turn the raw benchmark data into a few charts:

Takeaways

Compared with the “maximum throughput” benchmark in Part 1, this test removes one important variable by targeting the same request rate for both gateways.

Even under this controlled workload:

agentgateway sustained the full 3,000 QPS target, while LiteLLM averaged 2,466 QPS.
P99 latency was under 0.5 ms for agentgateway versus over 30 ms for LiteLLM.
agentgateway used approximately 26× less CPU on average.
Memory usage remained around 13 MB, compared to nearly 12 GB for LiteLLM.

Like the first benchmark, this test intentionally isolates proxy overhead by using a mock backend. It does not evaluate model inference latency or compare gateway features. If your application spends hundreds of milliseconds waiting for an LLM response, proxy latency becomes less significant.

However, if you’re building high-throughput AI infrastructure, serving many concurrent requests, or simply want a lightweight local gateway to manage all of your LLM providers, proxy efficiency matters. In this benchmark, agentgateway consistently delivered lower latency while using substantially fewer CPU and memory resources.

The complete benchmark scripts and raw results are available in the GitHub repository if you’d like to reproduce the numbers yourself.

Benchmarking Agentgateway vs LiteLLM

Fri, 26 Jun 2026 00:00:00 +0000

One of the coolest features in the agentgateway 1.3 release is Virtual Models. It made me realize I could use agentgateway locally to manage all of my LLM API keys while also getting visibility into my LLM usage and costs.

I’ve heard good things about LiteLLM for quite a while, so I decided to compare its performance with agentgateway. My goal is to find a lightweight gateway that can handle very high throughput with minimal latency. (I occasionally ask LLMs to generate videos, which results in a lot of API traffic.)

Instead of comparing features, I wanted to answer a simple question:

How much overhead does each proxy introduce under load?

Specifically, I measured:

Throughput (QPS)
Request latency
CPU utilization
Memory usage

Test setup

The benchmark uses a very simple architecture. A mock LLM server immediately returns a fixed response so the benchmark measures proxy overhead rather than model inference time.

Fortio generates as much traffic as possible against each gateway.

fortio (bt) ──► litellm :4000 ───────┐
├──► mock-server (hyper-server) :8081
fortio (bt) ──► agentgateway :4001 ──┘

I ran the benchmark using the default configuration:

./scripts/run-benchmark.sh

The benchmark uses:

32 concurrent connections
Maximum possible throughput (unlimited QPS)
1 KB request payloads
3-second benchmark duration

Results

Throughput & Latency

Gateway	Throughput	P50	P90	P99
agentgateway	36,933 QPS	0.831 ms	1.533 ms	1.970 ms
LiteLLM	3,198 QPS	7.076 ms	17.986 ms	32.192 ms

agentgateway handled over 11× more requests per second while maintaining sub-2 ms P99 latency.

CPU & Memory

Gateway	Avg CPU	Peak CPU	Avg Memory	Peak Memory
agentgateway	105%	243%	22 MB	29 MB
LiteLLM	331%	1075%	11.8 GB	11.8 GB

The memory difference was the biggest surprise. During this benchmark, LiteLLM consumed nearly 12 GB of RAM, while agentgateway stayed below 30 MB.

Raw benchmark output

./scripts/run-benchmark.sh
==> Run ID: 20260626-120716
==> LiteLLM workers: 18
...
Running fortio to litellm at 0 QPS for 3s and 32 connections...
qps: 3198.48qps p50: 7.076ms p90: 17.986ms p99: 32.192ms
...
Running fortio to agentgateway at 0 QPS for 3s and 32 connections...
qps: 36933.62qps p50: 0.831ms p90: 1.533ms p99: 1.970ms
...
DEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99
litellm,fortio,0,32,3,1104,10186,3198.48qps,7.076ms,17.986ms,32.192ms
agentgateway,fortio,0,32,3,1104,110831,36933.62qps,0.831ms,1.533ms,1.970ms
...
CONTAINER,SAMPLES,AVG_CPU%,PEAK_CPU%,AVG_MEM,PEAK_MEM
perf-agentgateway,4,104.80%,243.02%,22.38MiB,28.79MiB
perf-litellm,4,330.79%,1075.38%,11.81GiB,11.81GiB
perf-mock-server,4,24.52%,81.05%,3.43MiB,3.84MiB

Visualized results

I asked Cursor to turn the raw benchmark data into a few charts:

Takeaways

For this benchmark, agentgateway introduced significantly less proxy overhead than LiteLLM:

~11.5× higher throughput
Much lower (~10×) latency across all percentiles
~500× lower memory usage
Lower CPU utilization

This benchmark intentionally isolates proxy performance by using a mock backend, so it doesn’t measure real LLM inference latency or feature completeness. If your workload is dominated by model inference, the differences will be less noticeable. However, if you’re building high-throughput AI services or running a local gateway that handles many concurrent requests, proxy overhead becomes much more important.

Based on these results, I’m going to switch my local setup to agentgateway and use it to manage all of my LLM traffic.

Because each gateway handled a different workload in this test, I followed up with Part 2 using a fixed 3,000 QPS target for both gateways to make the comparison more apples-to-apples.

The complete benchmark scripts and raw results are available in the GitHub repository if you’d like to reproduce the numbers yourself.

agentgateway Standalone: A Cost & Tokenomics Dashboard in One Command

Wed, 24 Jun 2026 00:00:00 +0000

Introduction

You’re routing LLM traffic through a gateway. But do you actually know what it costs? Not the rough monthly invoice from your provider — the real breakdown. Which model burned the most tokens last night? Which user is driving 80% of your spend? Which provider is quietly eating your budget?

agentgateway answers those questions out of the box. Every request that flows through the proxy is priced against a per-model rate catalog and surfaced in a built-in Costs and Analytics dashboard. No external observability stack, no Prometheus, no Grafana — just the standalone binary.

This guide gets you from zero to a fully populated tokenomics dashboard in a single command. We’ll use a Docker-based demo that seeds 5,000 simulated requests across 7 days, so the dashboard has something interesting to show you the moment it boots — then we’ll send real traffic through it and watch it get priced live.

Why This Matters

Cost visibility is the FinOps story for AI. As soon as more than one team, agent, or app starts calling LLMs through shared infrastructure, “what did this cost and who spent it?” becomes a board-level question. agentgateway answers it at the gateway layer, which means:

Per-model pricing — every request is priced against a rate catalog (input, output, and cache token rates), so dollars show up next to tokens automatically.
Group by anything — slice spend and tokens by model, provider, user, group, or user agent (Cursor, Claude Code, openai-python, etc.).
Zero application changes — your apps just point at the gateway. The accounting happens in the proxy, not in your code.
One binary — the dashboard ships inside agentgateway. There’s no separate metrics pipeline to stand up for basic cost visibility.

What You’ll Build

 ┌──────────────────────────────┐
┌──────────────┐ │ agentgateway │
│ Your apps / │ /v1/chat/ │ ┌────────────────────────┐ │ ┌────────────┐
│ agents /curl │───completions──────▶│ │ LLM proxy (port 4000) │──┼─────▶│ OpenAI │
└──────────────┘ │ └───────────┬────────────┘ │ └────────────┘
│ │ priced per │
│ │ model catalog │
│ ┌───────────▼────────────┐ │
┌──────────────┐ localhost:15000 │ │ Admin UI + Dashboard │ │
│ Your browser│────────────────────▶│ │ Costs / Analytics │ │
└──────────────┘ │ └───────────┬────────────┘ │
│ │ │
│ ┌───────▼───────┐ │
│ │ SQLite data.db │ │
│ │ request_logs │ │
│ └───────────────┘ │
└──────────────────────────────┘

agentgateway proxies LLM traffic on port 4000 and serves its admin UI and dashboards on port 15000. Every request is written to a SQLite database (data.db) and priced using a model catalog (base-costs.json). The mock generator writes to the same request_logs schema, which is why the dashboard is populated before you send a single real request.

Prerequisites

Docker installed and running
curl
An OpenAI API key
Python 3.11+ (or uv) — used by the mock-data generator

Quick Start: One Command

Clone the demo and run the setup script:

git clone https://github.com/sebbycorp/agentgateway-demos.git
cd agentgateway-demos/00-standalone-latest

export OPENAI_API_KEY='sk-...'
./setup.sh

That’s it. Open http://localhost:15000/ui/ and head to the Costs and Analytics sections — they’re already full of data.

What the script does

setup.sh is a single-command bootstrap. Under the hood it:

Runs preflight checks — verifies Docker is installed and the daemon is running, curl is available, OPENAI_API_KEY is set, and that uv or Python 3.11+ is present.
Downloads the mock-data generator (gen-mock-logs.py).

Generates a SQLite database with simulated fleet traffic:

gen-mock-logs.py --replace --requests 5000 --days 7 -o data/data.db

Writes config.yaml pointing the gateway at the database and the model rate catalog.
Pulls the agentgateway image (cr.agentgateway.dev/agentgateway:v1.3.1) and removes any previous demo container.
Seeds a named Docker volume (agw-cost-demo-data) with the generated database.

Launches the container, binding ports to loopback only:

127.0.0.1:4000:4000 # LLM proxy
127.0.0.1:15000:15000 # admin UI + dashboards

Why loopback only? The proxy port carries your API credentials. Binding to 127.0.0.1 keeps the demo off your network. Don’t expose these ports without locking down auth and CORS first.

Tuning the dataset

Want a bigger or smaller demo dataset? Override the REQUESTS and DAYS environment variables before running setup:

REQUESTS=20000 DAYS=30 ./setup.sh

Exploring the Dashboard

Open http://localhost:15000/ui/ and click Analytics. By default it shows total traffic over the last 24 hours — token volume per hour with a running tally of cost, tokens, and calls.

The real power is in Group by. Switch it to Provider and the same traffic splits out by backend — here OpenAI dominates with ~13.3M tokens, followed by Anthropic, Google, and Bedrock. The breakdown table underneath ranks every provider by token consumption.

Switch Group by to User and you get per-person accounting — exactly the view you need when you’re trying to figure out who’s driving spend. Each bar in the time series is stacked by user, and the breakdown ranks them by tokens consumed.

You can group by Model, Provider, User, Group, or User agent (Cursor, Claude Code, openai-python, codex, bifrost, and more), and switch the Measure between tokens and cost. The Costs page focuses the same data on dollars, and Export lets you pull the underlying numbers out for reporting.

How It Works: The Generated Config

The setup script writes a config.yaml that wires everything together. Here are the pieces that matter:

config:
 adminAddr: "0.0.0.0:15000" # admin UI + dashboards (reachable from host)
 database:
 url: "sqlite:///data/data.db" # /data is the mounted ./data dir in the container
 modelCatalog:
 - file: /base-costs.json  # per-model rates so every request is priced

llm:
 port: 4000
 policies:
 cors: # demo-only: wildcard CORS. Safe because the port
 allowOrigins: ["*"] # is loopback-bound. Restrict this for real use.
 allowHeaders: ["*"]
 allowMethods: ["GET", "POST", "OPTIONS"]
 models:
 - name: "openai/gpt-4.1"
 provider: openAI
 params:
 model: gpt-4.1
 apiKey: "$OPENAI_API_KEY"
 - name: "openai/*" # fallback: cheaper nano model
 provider: openAI
 params:
 model: gpt-4.1-nano
 apiKey: "$OPENAI_API_KEY"

Three things make the dashboard work:

database.url — the admin UI’s Costs/Analytics pages read from this SQLite database. The mock generator writes the same request_logs schema, so generated traffic and real traffic land in one place.
modelCatalog — base-costs.json holds per-model input/output (and cache) token rates. This is what turns raw token counts into dollars.
models — two routes: an explicit openai/gpt-4.1 and a wildcard openai/* that falls back to the cheaper gpt-4.1-nano.

Send Real Traffic

The mock data gets you a populated dashboard, but the gateway is live — send it a real request and watch it get priced alongside the simulated traffic:

curl -X POST http://localhost:4000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "openai/gpt-4.1",
 "messages": [{"role": "user", "content": "Hello from agentgateway!"}]
 }'

Refresh the Analytics page and your request shows up — tokens counted, cost calculated against the model catalog, attributed to the model and provider. Every real call from here on is accounted for the same way.

Manual Installation

Prefer to run the binary directly instead of the Docker demo? You have three options:

# 1. Automated installer
curl -sL https://agentgateway.dev/install | bash

# 2. Download a platform-specific binary from the GitHub releases page
# https://github.com/agentgateway/agentgateway/releases

# 3. Run via Docker with your own config mounted
docker run --rm \
 -p 127.0.0.1:4000:4000 \
 -p 127.0.0.1:15000:15000 \
 -v "$(pwd)/config.yaml:/config.yaml" \
 cr.agentgateway.dev/agentgateway:v1.3.1 --file /config.yaml

Point it at a config.yaml like the one above, and the proxy listens on port 4000 with the admin UI on 15000. From there it’s the same dashboard — minus the pre-seeded mock data.

Cleanup

When you’re done, tear the demo down with the included script:

./destroy.sh

This stops and removes the container and the named volume.

Wrapping Up

Cost and token visibility is one of those things you don’t realize you’re missing until a bill lands. agentgateway puts it right in the box: per-model pricing, a built-in dashboard, and grouping by model, provider, and user — no external observability stack required. The Docker demo gets you a populated dashboard in one command so you can see exactly what it looks like before pointing real traffic at it.

Resources

Agentgateway v1.3.0: LLM Consumption, Reimagined

Wed, 17 Jun 2026 00:00:00 +0000

Agentgateway v1.3.0 is here — and it’s the biggest leap yet in how you consume LLMs. We rebuilt the experience around the way developers actually work: one endpoint, the model in the request body, and every concern that used to demand a separate tool — cost, security, guardrails, observability — now handled right at the gateway. No custom proxies, no SDK sprawl, no waiting on the next provider release to catch up.

Here’s what landed:

A brand-new, purpose-built UI — native LLM, MCP, and Traffic views
AI Cost & Analysis — token and dollar cost, fully attributed, in logs, traces, metrics, and the UI
Virtual models — weighted, conditional, and failover routing
Reusable providers & guardrails — define once, reference everywhere
13 new LLM providers — Mistral, Hugging Face, Cohere, and ten more

All of it aligned with the model-based routing pattern every developer already expects. This is what a true AI-native gateway feels like.

Try it now →

The new UI — built for the LLM-first world

We didn’t just refresh the UI. We rebuilt the mental model around how you actually consume LLMs and MCP services. Pick the capabilities you need on first launch and you’re off:

The new onboarding flow — enable LLM, MCP, and API capabilities and get going in seconds

Everything is now organized into three clean, native views:

LLM — models, providers, policies, guardrails, costs, virtual API keys
MCP — servers, tools, resources, auth
Traffic — the classic Gateway API experience you already know

Onboard providers and models in seconds. Point an incoming model match at a provider and you’re routing — add provider-backed models or virtual models from the same screen.

Onboard a model — map an incoming model match to a provider and save

Apply policies without leaving the LLM layer. CORS, API keys, external authz, OIDC, JWT, and rate limiting are all configurable per model and visible at a glance.

Configure LLM policies and per-model behavior in one place

Add guardrails with a click. Built-in detectors, custom regex, webhooks, OpenAI moderation, Bedrock Guardrails, and Google Model Armor — request and response guards, straight from the UI.

Add request and response guardrails — built-in detectors, custom regex, webhooks, and provider guardrails

Configure once at the LLM layer, and it just works everywhere.

New: AI Cost & Analysis — every token, every dollar, attributed

No more guessing. No more scraping logs to figure out your burn rate. Agentgateway v1.3.0 turns every request into a measured, attributed, exportable data point — giving you full token and dollar analysis across everything flowing through the gateway.

Configure cost rates per model, or import the official provider tables
Every request calculates its exact cost and token counts (input, output, total)
Token and cost data is surfaced in logs, traces, metrics, the UI, and agctl

But the real unlock is attribution. Agentgateway doesn’t just tell you what you spent — it tells you who spent it, on what, and through which tool. The Analytics view lets you slice token usage and dollar cost across every dimension that matters, and measure by tokens or cost:

Per user — see exactly which developer or service account is driving spend, down to the individual user@example.com.
Per team — roll usage up by group so you can charge back to the right cost center and spot the team that just 10x’d its Opus bill.
Per coding tool / agent — break spend down by user agent, so you can compare what Claude Code, Cursor, GitHub Copilot, and your own autonomous agents are actually costing you across the org. When finance asks “is the Cursor rollout worth it?”, you have the number.
Per model — claude-opus-4-7 vs claude-sonnet-4-6 vs gpt-5.1, side by side, so you know where the tokens are really going.
Per provider — compare OpenAI, Anthropic, Bedrock, and the rest at a glance to inform routing and contract decisions.

Mix and match the filters — “how much did the support team spend on Claude through Cursor this week?” is now a two-click question, not a data-engineering project. And it’s not just LLMs: because LLM, MCP, and A2A traffic all flow through the same gateway, you get one consistent, visual picture of how your entire agentic stack is being consumed — tokens, calls, and dollars, side by side.

Group token usage and cost by model, user, team, provider, or coding tool — measure by tokens or dollars, then export

Need the numbers somewhere else? Export the report straight from the Analytics view and hand it to finance, drop it into a chargeback spreadsheet, or feed it into your own BI pipeline.

Because attribution lives at the gateway, you can also act on it with the same policy engine you already use for auth and rate limiting: set budgets, alert on spend, enforce hard quotas per team, or automatically route a cost-sensitive consumer to a cheaper model.

And it’s not just aggregate dashboards. The Logs view captures LLM, MCP, and A2A calls in one place — drill into any individual request and see exact duration, input/output tokens, dollar cost, and the user, group, and provider it was attributed to:

Per-request cost, timing, and token breakdown — LLM, MCP, and A2A calls in one Logs view

Virtual models — route smarter, not harder

Real model routing is rarely “send everything to one model.” You want to A/B test a new release against the incumbent, fall back to a cheaper model when the primary is throttled, or send long-context requests somewhere with a bigger window. Historically that meant routing logic baked into every client, or a custom proxy nobody wanted to own.

Virtual models move that decision into the gateway. A virtual model is a synthetic model — it has a name like any other, but instead of pointing at a single backend, it applies a routing strategy across several real models. Your client just sends the virtual model’s name in the request body; the gateway decides where each request actually goes.

One name in, the right backend out — the virtual model applies a weighted, failover, or conditional strategy per request

Three strategies ship in v1.3.0, with more planned:

Weighted — split traffic by percentage across models. Send 70% to your current production model and 30% to a candidate to A/B test quality or cost before you commit. Dial the split without touching a single client.
Failover — order a list of models by preference. If the primary returns errors or gets rate-limited, the gateway automatically retries the next one — so a provider hiccup degrades gracefully to a fallback instead of failing the request.
Conditional — branch on the request itself using CEL. Route by user tier, prompt length, headers, or any attribute the gateway can see — for example, send premium users to a frontier model and everyone else to a cheaper one.

The win is decoupling: routing policy lives in one place, owned by the platform team, and every consumer benefits at once. Want to flip the A/B split, add a fallback, or move a tier to a new model? Change the virtual model — no client changes, ever — and it takes effect everywhere instantly.

Reusability and a provider explosion

Reusable providers and guardrails — define a provider or guardrail once and reference it across many models. Managing 50 OpenAI-compatible models? Done.
Custom provider — reach providers without built-in support. More flexible than the OpenAI + base_url hack, and the recommended path for anything non-first-class.
13 brand-new first-class providers — Mistral, Hugging Face, Cohere, and ten more.
Streaming guardrails — guardrails now apply to streaming requests too, and webhook guardrails gain a failureMode (fail-open or fail-closed).
Shared guardrails in standalone — declare guardrails once as a top-level resource instead of repeating the block on every route.

This is the release where managing many models stops feeling like a spreadsheet nightmare.

Beyond the LLM layer — the whole platform leveled up

v1.3.0 isn’t only about LLMs. Here’s everything else that landed.

Request handling & extensibility

Request buffering — traffic policies can buffer request bodies before forwarding, so policies and extensions get full request-body access before backend selection.
External processing — richer processing-mode configuration, and ext_proc can now return an ImmediateResponse from the request- and response-body phases.
ExtMCP — MCP-aware external auth and external processing, so your services decide using MCP request context, not just generic HTTP metadata.

Authorization & auth

Authorization can now run in the pre-routing phase
Per-model LLM authorization — scope who can call which model
External-auth cache TTL as an expression, plus ext-authz caching
Credential-location expressions fixed and expanded; scheme derived from X-Forwarded-Proto

LLM gateway

Rerank request/response support across providers
Custom LLM providers for InferencePool backends
More precise per-model matching (prefers exact matches)
Bedrock: detect-passthrough, Application Inference Profile prompt cache, Anthropic beta-header allowlist, host override, URL-encoded model IDs, reasoning-signature replay
Anthropic: system messages and extra-high thinking
Local LLM: TLS and CORS support
Latency and throughput telemetry attributes on LLM requests

MCP

Okta as a first-class MCP authentication provider
Resource subscribe/unsubscribe, improved multiplexing and list behavior
Advertised prompt/resource/tool list-change capabilities, plus broader protocol-compliance fixes

And like the LLM layer, MCP gets its own native policy views — access, traffic shaping, and mutation, configurable per server:

MCP policies — authorization, CORS, JWT, rate limiting, transformations, and external processing for MCP traffic

TLS, networking & policy

Dynamic SSL certificates for Kubernetes listener TLS
Generalized backend TLS and backend references, with a new BackendReferenceGrantMode
Policy inheritance strategy configuration, and composable AI backend policies
Terminating inbound CONNECT, configurable admin interfaces (including UDS), AWS AssumeRole, custom AWS service names, and mTLS cert passthrough via CEL

CEL & agctl

New CEL helpers: url.encode/url.decode, more timestamp conversions, bit ops on bytes, jwt.rawToken, gRPC status on responses, expressions in direct responses, and CEL-based retry conditions
agctl gains proxy/controller log commands, a version command with mismatch checks, route groups in config output, and evicted-backend visibility

Operations & observability

Proxy timing measurements and a config-synchronization metric
Request and connection IDs for troubleshooting
Richer distributed traces — JSON mode, body snapshots, effective gateway/route policies, and raw-output file opening

A handful of fixes worth calling out: TCP route precedence, Gateway status when no listeners are valid, route-level OIDC cookie handling, capacity-weighted load balancing, backend eviction retries, and streaming-completion capture across Bedrock, Messages, and Responses API paths.

For the complete list of features and 50+ fixes, see the GitHub release notes.

Availability

Agentgateway v1.3.0 is available now. Grab the Docker images, Helm charts, and binaries from the GitHub release page, then follow the quick start guide for Kubernetes or standalone.

Contributors

This release happened because 41 contributors showed up between v1.2.1 and v1.3.0 — code, reviews, docs, bug reports, and the unglamorous CI work that keeps a fast-moving project on the rails. 21 of them made their first commit this cycle alone.

A special shout-out to the people who drove the most change this release:

…alongside everyone whose fixes, features, and feedback made the rest of the release possible. The full list of contributors is in the v1.3.0 release notes.

Get involved

Star the repo at github.com/agentgateway/agentgateway, join us on Discord, and come hang out at our community meetings.

agentgateway Joins AAIF as an Open Gateway for Agentic AI Infrastructure

Thu, 04 Jun 2026 00:00:00 +0000

Originally published by the Agentic AI Foundation.

AI systems increasingly resemble distributed systems. Agents invoke tools, models route across providers, and workflows span APIs, MCP servers, databases, and other agents. As these systems move from prototypes to production, organizations are finding that infrastructure designed for traditional web traffic lacks the governance, observability, routing, and security controls that agentic systems demand.

agentgateway addresses this gap. The project has joined the Agentic AI Foundation as the fourth hosted initiative under the Linux Foundation.

Overview

agentgateway is an open source gateway engineered for modern AI system architectures. It delivers a unified management layer for MCP traffic, Agent-to-Agent communication, LLM inference, REST APIs, and gRPC services through a consolidated operational interface.

Rather than deploying separate infrastructure for AI workloads, organizations leverage agentgateway to manage AI and conventional application traffic using identical security controls, observability systems, routing policies, and governance frameworks.

Key Capabilities

MCP and A2A support — routing and federation for agent interoperability protocols
Model independence — seamless LLM provider switching with open-weights model flexibility
Unified data plane — single gateway for HTTP, gRPC, LLM inference, and agent workloads
Security controls — JWT authentication, API key validation, RBAC, external authorization, mTLS, CORS, and malicious tool protections
Built-in observability — metrics, tracing, and access logs for AI workflows
MCP virtualization — consolidate multiple MCP tool servers with tool-level access policies
Declarative policy via CEL — dynamic configuration using Common Expression Language
Governance features — rate limiting, content-based routing, prompt guards, and budget controls
Platform-agnostic deployment — bare metal, VMs, containers, and Kubernetes
Dynamic configuration — xDS-based updates without service interruption
High-performance design — Rust implementation for low-latency operations

Ecosystem Impact

AAIF provides neutral governance for agentic AI standards, protocols, and open source initiatives. agentgateway strengthens this ecosystem by establishing shared infrastructure for AI traffic, agent workflows, interoperability, security, and governance.

David Soria Parra, AAIF Technical Committee Chair, emphasized: “agentgateway helps fill that gap with an open, high-performance platform designed specifically for these emerging workloads.”

Lin Sun, Solo.io Head of Open Source and agentgateway contributor, noted the initiative reflects “open connectivity and interoperability” principles with “open collaboration and neutral governance.”

The project operates under Apache 2.0 licensing, with 300+ active contributors across 60+ organizations including CoreWeave, Red Hat, Solo.io, Adobe, Salesforce, Amdocs, and Microsoft.

Getting Started

Documentation: agentgateway.dev
Code: GitHub repository
Getting Started Guide: Available on the official site
Community: Discord channel
Meetings: Weekly community gatherings (details in the repository README)

Designing agentgateway: A Unified High-Performance Gateway for AI and API Traffic

Thu, 04 Jun 2026 00:00:00 +0000

Originally published by the Agentic AI Foundation.

When we started building agentgateway at Solo.io, one of the first questions we asked ourselves was whether the world really needed another gateway instead of simply reusing an existing reverse proxy like Envoy.

The answer became obvious pretty quickly.

Not because existing API gateways are broken. And not because AI traffic somehow replaces traditional application traffic.

The reason is that organizations deploying AI systems are running into a new category of operational problems that existing infrastructure was never specifically designed to address.

AI systems create new infrastructure concerns

As agents become more capable, they stop looking like isolated chatbot experiences and start behaving more like distributed systems.

They call tools. They route requests across APIs and models. They coordinate workflows across multiple services. They interact with MCP servers, LLM providers, databases, and internal platforms.

And suddenly the infrastructure questions become much bigger than simple request routing.

How do you govern which tools an agent can access?
How do you apply consistent authentication and authorization policies across AI systems and traditional services?
How do you observe what agents are actually doing across complex workflows?
How do you rate limit model usage, apply routing policies, or enforce organizational controls around AI traffic?
How do you minimize impact between AI agents and MCP servers with fast evolving MCP protocols?
How do you gain security, governance and visibility into the emerging context layer (“layer 8”) that is becoming increasingly important for AI traffic?

These are not theoretical questions anymore. Teams deploying AI systems are dealing with them right now.

Why separate AI gateways and API gateways becomes painful

One of the earliest architectural decisions we made was to avoid creating a completely separate infrastructure stack for AI systems.

At first glance, splitting them apart sounds reasonable. Traditional APIs and AI workloads appear different enough to justify separate operational layers.

But in practice, organizations running agents are also running the APIs, services, and infrastructure those agents depend on.

That means platform teams quickly end up duplicating operational concerns across multiple systems:

separate policy models
separate authentication configurations
separate observability pipelines
separate routing infrastructure
separate governance controls

The complexity compounds fast.

For example, an internal developer platform might expose several MCP servers for documentation search, ticket creation, deployment workflows, and database access. Without a shared gateway layer, each team may need to configure authentication, authorization, audit logging, rate limits, and routing separately. agentgateway gives platform teams a centralized place to manage those controls while still allowing individual tools and services to evolve independently.

We believed the better approach was unification.

agentgateway was designed as a unified gateway control plane and proxy data plane that can handle HTTP, gRPC, MCP, A2A, and LLM traffic together through the same operational surface. That means organizations can manage AI and non-AI traffic using the same infrastructure patterns instead of standing up parallel systems for each.

Tool federation and protocol-aware routing

One of the challenges that emerges quickly in real AI deployments is fragmentation.

Different teams expose different MCP servers. Organizations adopt multiple model providers. Agents interact with internal APIs, external services, and specialized tooling spread across environments.

From both an operational and security perspective, managing those integrations individually becomes difficult.

agentgateway introduces a federation layer that allows organizations to aggregate and route traffic across tools, models, and services while applying centralized policy enforcement and visibility controls.

Clients can interact through a unified endpoint while administrators retain control over authentication, authorization, observability, and routing behavior.

This becomes increasingly important as interoperability protocols like MCP and A2A continue gaining adoption across the ecosystem.

Why we chose Rust

We didn’t create agentgateway entirely from scratch.

Over the past three years, we’ve been building Istio ambient service mesh within the community. A key component of Istio ambient is ztunnel, a Rust-based, purpose-built lightweight proxy designed to handle the secure overlay layer.

We applied many of the lessons learned from building ztunnel, as well as years of experience operating Envoy at scale, to agentgateway, enabling us to create an AI-native proxy optimized for performance, security, and operational simplicity.

Like ztunnel, we built agentgateway in Rust because performance and memory safety are non-negotiable for this kind of system.

Rust has a strong history of success in high performance, low resource utilization applications, especially in network applications (including service mesh). We built agentgateway on top of Tokio and Hyper, two extensively battle-tested libraries for asynchronous networking, along with Tonic, cel-rust, and other core ecosystem components.

Performance

When evaluating infrastructure components like gateways, performance and scalability are critical.

Common gateway performance metrics include:

Throughput
Latency
CPU and memory utilization
Route propagation time
Error rates
The ability to safely handle route updates without downtime

Equally important is how the system behaves at scale: tens of thousands of services, MCP servers, and routes operating simultaneously within highly dynamic distributed systems.

agentgateway uses an xDS control plane architecture that allows dynamic configuration updates without restarting the data plane. Routes, policies, integrations, and backend services can evolve continuously while traffic continues flowing.

Using traffic performance as one example, agentgateway achieves approximately 500k QPS with 512 connections in our benchmark testing, outperforming peer proxies under similar conditions.

In another benchmark, agentgateway maintained less than 0.2 ms P99 latency at 30k QPS with 512 concurrent connections.

For a deeper dive into methodology and additional metrics beyond traffic performance, check out John Howard’s Gateway API benchmark v2.

Why we donated agentgateway to AAIF

From the beginning, we wanted agentgateway to live under vendor-neutral governance.

After creating the project in March 2025, we initially donated it to the Linux Foundation on August 25, 2025 because neutral governance was important to both contributors and users.

At the same time, we continued searching for a foundation more specifically aligned with agentic AI infrastructure. When the Agentic AI Infrastructure Foundation (AAIF) launched, it became clear that it was a strong long-term fit.

AAIF provides a neutral, open foundation to help critical AI infrastructure evolve transparently and collaboratively while accelerating adoption of open-source AI projects.

agentgateway complements existing AAIF projects such as MCP and Goose by acting as the connective layer between:

Agent-to-LLM interactions
Agent-to-MCP interactions
Agent-to-agent interactions

It provides the security, governance, and observability enterprises need to confidently adopt AI agents, MCP servers, and LLM-based systems.

On April 8, we submitted the agentgateway proposal to AAIF through the project proposal process. The project was approved by the Technical Committee on May 13 and by the Governing Board on May 21 as a Growth-stage project.

Tremendous Growth

Since February, we’ve seen rapid adoption of agentgateway.

Weekly downloads grew from approximately 100,000 per week to more than 1 million per week, surpassing 7 million total downloads.

We also use agentgateway extensively at Solo.io to mediate both LLM and MCP traffic, giving us consistent security, governance, and observability across these systems.

In parallel, we’ve been working closely with organizations including Microsoft, Apple, Adobe, Amdocs, T-Mobile, and Expedia, along with many other enterprises adopting agentgateway.

agentgateway has also been adopted by Istio as a data plane option for AI gateway use cases.

What’s next

As agentic systems move from experiments into production environments, the community will need shared infrastructure patterns for routing, policy enforcement, observability, and interoperability.

We want agentgateway to become a unified gateway layer to secure, connect, and observe agentic and cloud native workloads.

Over the next 12 months, we plan to continue expanding the project through deeper integrations, broader protocol support, and continued collaboration with the open-source community.

Some planned areas include:

Enhancing our UI to include historical analytics and request information, as well as broader support for AI integrations
Expanding our inference workload support through integration with LLM-d and vLLM Semantic Router
Continuing to engage with, and implement, new MCP proposals such as Stateless MCP
Expanding MCP functionality with richer guardrails and cost optimization features such as progressive disclosure and code mode
Integration with Agent Client Protocol (ACP)
Publishing more production case studies and architecture patterns
Expanding internationalization and community-driven translation workflows
Continuing collaboration with the Kubernetes community on agentic networking and AI Gateway APIs

With agentgateway now part of AAIF, we welcome contributors, users, and platform teams to help shape the roadmap in the open under neutral governance.

If you are building AI systems, operating MCP infrastructure, or thinking about the operational maturity required for agentic workflows, we would love to collaborate with you.

Explore the docs and get started today.
Star and contribute on GitHub.
Engage with us in the community Discord or an upcoming community meeting.

Agentgateway v1.2.0: we're shipping like the agents are watching

Thu, 14 May 2026 00:00:00 +0000

The agent ecosystem isn’t waiting for anyone. New models drop weekly. New protocols land monthly. New attack surfaces, new auth schemes, new deployment topologies — every quarter the ground shifts under whatever you built last year. A gateway that ships on a yearly cadence is a gateway that’s already obsolete.

So we don’t. Agentgateway ships on a monthly cadence — v1.2.0 lands roughly a month after v1.1.0, and the next minor will land a month after this one. The release train moves at the speed of the ecosystem, not the speed of a planning offsite.

v1.2.0 packs twenty-plus new capabilities into one release: route delegation, backend external auth with credential exchange, conditional policies with CEL, an honest-to-god debugger CLI, automatic xDS TLS, PROXY protocol, locality-aware failover, post-quantum key exchange, and a stack of LLM gateway upgrades that finally make Azure and Copilot first-class.

Here are the four that change how you’ll run agentgateway day to day — with code, context, and the reason we built them.

Heads up: one breaking change before you upgrade

Before you update: xDS TLS is no longer a boolean. The Helm value moved from controller.xds.tls.enabled: true to controller.xds.mode: tls | plaintext | either.

The Helm chart now defaults to tls, and the controller can manage its own xDS certs (more on that below).

If you’ve got automation pinning the old key, swap it before you helm upgrade.

1. Route delegation: stop being the YAML bottleneck

Every platform team eventually hits the same wall: app teams keep filing PRs against a centralized gateway config because they need a new path, a new header rewrite, a new backend. The platform team becomes the bottleneck. Velocity dies. Tickets pile up.

Route delegation kills that pattern. A parent HTTPRoute, owned by platform, hands off a sub-path to child routes in a different namespace, owned by the app team. The children show up, register themselves under the parent’s tree, and the gateway figures out the rest.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: parent
 namespace: agentgateway-system
spec:
 hostnames:
 - delegation.example
 parentRefs:
 - name: agentgateway-proxy
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /anything/team1
 backendRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: "*"
 namespace: team1
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: child-team1-foo
 namespace: team1
spec:
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /anything/team1/foo
 backendRefs:
 - name: httpbin
 port: 8000

Platform owns the trunk. Teams own their branches. Nobody opens a PR against gateway.yaml ever again. This is the multi-tenant story a lot of you have been asking for, and it’s straight Gateway API — no proprietary CRDs in the hot path.

For a walkthrough, see the route delegation docs.

2. Backend external auth: token exchange without the sidecar tax

Here’s the pattern everyone ends up building eventually: a user logs in, hits your gateway with a JWT, and the gateway needs to call three different upstreams — each of which wants a different credential. One wants a GCP service account token. One wants a partner-issued API key. One wants a short-lived OAuth token scoped to that user.

Until now, the “right” answer was a sidecar, an envoy filter chain held together with hope, or a homegrown proxy in front of your proxy. That’s done.

External auth can now run as a backend policy after backend selection. The gateway knows exactly which upstream is about to be called, calls out to your token-exchange service, and can place the returned credential where that backend expects it — header, query param, or cookie. You configure it once. The gateway handles the per-backend dance.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: backend-token-exchange
 namespace: agentgateway-system
spec:
 targetRefs:
 - group: agentgateway.dev
 kind: Backend
 name: partner-api
 traffic:
 extAuth:
 backendRef:
 name: token-exchange-svc
 namespace: agentgateway-system
 port: 9000
 grpc: {}

Combine it with the new credential-location overrides — JWT, basic auth, API key, and backend auth can now read from or write to headers, query params, or cookies — and explicit Secret-backed GCP credentials, and the awkward “auth proxy in front of the auth proxy” pattern just collapsed into one CRD.

3. Conditional policies: one block, many code paths

You’ve all built this monstrosity: three near-identical rate-limit policies, gated by header matchers on the route, because the existing API made you express “if write, do X; else do Y” by duplicating the entire policy block. Or worse — you punted to an ext-proc service just to get an if statement.

Now policies branch on CEL directly. External auth, transformations, rate limiting, ext-proc, and direct responses all support conditional: lists. The gateway walks them top-down, applies the first match, and falls back to the unconditional tail if nothing hits.

Here’s a real one — strict limits for writes, looser limits for everything else:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: conditional-ratelimit
 namespace: default
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: my-route
 traffic:
 rateLimit:
 conditional:
 - condition: request.method == "POST" || request.method == "PUT" || request.method == "DELETE"
 policy:
 local:
 - requests: 10
 unit: Minutes
 - policy:
 local:
 - requests: 100
 unit: Minutes

That’s the whole pattern. One policy, two paths, zero duplication. And CEL means you can branch on anything in the request — headers, path, JWT claims, even response state for response-side policies.

For details, see the conditional policies docs.

4. agctl: the debugger you’ve been asking for

Real talk: debugging a gateway in production has historically been a mix of kubectl logs, hope, and human pattern matching. Envoy’s admin interface gets you part of the way. The rest is detective work.

The new experimental agctl CLI changes that. Two commands you’ll reach for daily — and both auto-discover the proxy pod and set up the port-forward to the admin endpoint, so there’s no kubectl port-forward dance every time.

# What did the proxy actually load? Binds, listeners, routes, backends, policies.
agctl config all
# → renders the running runtime config in table, JSON, or YAML.

agctl config backends
# → just the active backends, with health scores, request counts,
# and observed latency.

# Trace a request end-to-end through filters, policies, and the upstream
agctl trace
# → step-by-step record of how the proxy handled a request:
# matched route, applied policies, selected backend, response status.
# Inject your own request, or watch traffic from real clients.

agctl config is the one most teams hit first. Your HTTPRoute reports Accepted: true, but traffic isn’t doing what you expect — and now you can see exactly what the proxy loaded instead of guessing from CRD status.

Then when somebody pings you with “the gateway is returning 502s on /v1/checkout,” you stop guessing there too. You run agctl trace, you see the JWT failed validation because the JWKS cache went stale, and you fix it before the standup ends.

Pair it with the proxy’s admin endpoint on port 15000 — /config_dump, /debug/trace, /logging for live log-level changes, and /debug/pprof/{profile,heap} for CPU and heap profiles — and the gateway stops being a black box. See the debug guide for the full playbook.

Everything else that landed (it’s a lot)

Automatic xDS TLS. Controller mints a local CA, issues short-lived serving certs, rotates them. Bring your own if you prefer. Either way, no more pre-creating xDS certs.

Networking & TLS. PROXY protocol v1/v2 (strict or optional). Locality-aware load balancing and failover for multi-zone and multi-region. HBONE gateway tunnel protocol for Istio ambient. Listeners can now use Istio workload certs for simple or mutual TLS. Unix Domain Socket backends. Post-quantum TLS via X25519_MLKEM768. Max connection duration on HTTP listeners. HTTP/2 pooling that no longer bottlenecks on a single connection.

LLM gateway. Azure OpenAI and Azure AI Foundry are now first-class providers. Copilot auth. Gemini Responses API. Custom path prefixes across every provider — Gemini, Vertex, Bedrock, Azure. Azure Content Safety guardrails. Bedrock guardrail masking. OpenAI requests normalize max_tokens to max_completion_tokens automatically.

MCP. Idle session TTL. Cleaner stateless lifecycle. ListResourcesRequest works across multiplexed targets.

Policy targeting. AgentgatewayPolicy now accepts selector-based targets and can attach to ListenerSet and InferencePool. Custom Prometheus metric labels via CEL.

Performance. New allocator → higher throughput, lower RSS. Plus a long tail of fixes across CEL, JWKS caching, header/:authority alignment, hop-by-hop header stripping, and Gateway status update churn.

For the complete list, see the release notes.

How fast are we shipping?

We shipped this in months, not quarters. The roadmap moves on the same cadence as the agent ecosystem it serves — because the alternative is being the slow part of someone else’s stack, and that’s not a future we’re interested in.

Availability

Agentgateway v1.2.0 is available for download on GitHub.

To get started with agentgateway, check out our getting started guide for standalone or Kubernetes.

Contributors

This release happened because 31 contributors showed up across 166 commits between v1.1.0 and v1.2.0 — code, reviews, docs, bug reports, and the unglamorous CI work that keeps a fast-moving project from face-planting.

Top contributors by commit count this cycle: @howardjohn, @npolshakova, @stevenctl, @danehans, @markuskobler, @filintod, and @syn-zhu — alongside two dozen more whose fixes, features, and feedback made the rest of the release possible.

The full list of contributors is in the v1.2.0 release notes.

Get involved

Star the repo at github.com/agentgateway/agentgateway, join us on Discord, and come hang out at our community meetings.

Happy 1st birthday, agentgateway!

Wed, 25 Mar 2026 00:00:00 +0000

On this day in 2025, we created agentgateway after first carefully evaluating Envoy Proxy. As a company deeply invested in Envoy, we initially didn’t want to start a new project, but we needed to keep pace with the rapid evolution of AI. Building on our success developing the new proxy for Istio Ambient, we ultimately decided to create agentgateway from the ground up.

Rapid adoption among users

Since we donated the project to the Linux Foundation in August 2025, we have witnessed continued strong growth of the project. In the most recent quarter alone, we’ve reached 1 million image pulls of agentgateway.

We have over 2,000 GitHub stars on agentgateway’s main repository, with continuing growth. Thank you everyone who starred the agentgateway repo.

Scroll through birthday wishes from some of our users and contributors.

" As platform teams start running AI agents alongside their services, the communication layer becomes the new control plane. Agentgateway gets this right. It brings the policy, observability, and traffic management primitives the cloud native community already trusts to agent-to-agent communication. This is infrastructure the cloud-native ecosystem needs. "

Joseph Sandoval

Platform Product Manager, Adobe

" Happy Birthday, Agentgateway! While the world was obsessed with the magic of AI agents, this project was busy building the essential guardrails and observability needed to make them prod ready. It's the unsung hero of secure agent communication. "

Chris Matcham

Senior Platform Engineer, Helcim

" At telco scale, an agentic system requires an AI Gateway by design. It acts as the proxy for traffic management and policy enforcement across A2A, MCP, and LLM interactions. We’re excited to see this project maturing under the Linux Foundation, reinforcing the importance of Kubernetes-native agentgateway for secure, portable, and scalable deployments across platforms. "

Roi Dayan

VP R&D, Data & AI, Amdocs

" We use agentgateway internally for all of our engineers’ LLM usage. It’s made onboarding faster and easier while giving engineers access to more providers than ever before, all from one centralized place. The improved visibility, user-level metrics, and rich telemetry have helped us drive increased AI usage while identifying optimization opportunities to keep costs down. "

Shane O’Donnell

VP of Engineering, Solo.io

Amazing diversity of contributors and partners

The LFX Insights provide project insight for many open source projects including agentgateway, and I was pleased to see the project currently has 179 active contributors:

Since we created the project, we are thrilled to witness contributions from many companies including but not limited to Microsoft, Apple, Alibaba, AWS, Adobe, Huawei, Amdocs, Cisco, and Salesforce.

Scroll through some thoughts from leaders and contributors in our community.

" Agentgateway's first anniversary is an exciting milestone for the open source AI agent community. In its first year, the project has helped convene contributors and adopters around open approaches to AI agent infrastructure, with support for protocols such as A2A and MCP and growing participation from across the ecosystem. It is encouraging to see this level of collaboration around technologies intended to support agent-to-agent, agent-to-tool, and agent-to-LLM interactions. As the ecosystem continues to develop, open, interoperable, and community-driven infrastructure will play an important role in supporting innovation. "

Matt White

CTO of AI at the Linux Foundation

" Agentgateway is evidence that we really can just build things. Its native support for cloud native AI scenarios are best-in-class, and we're excited by the work of two Istio maintainers to bring agentgateway into the Istio project, evolving the project towards an agentic mesh. "

Keith Mattix

Principal Software Engineer Lead, Microsoft

" The agentgateway open-source project enables a secure, observable MCP proxy for us to confidently deploy the Community AI platform engineer (CAIPE.io) at Outshift by Cisco. It's been a great collaboration so far and I wish the project continued success. "

Hasith Kalpage

Director, Platform Engineering & CISO, Outshift by Cisco

" As a developer building agentic systems, I have found that bridging the gap between local prototypes and secure, production-grade deployments is typically a massive hurdle, and this is exactly where agentgateway shines. Instead of forcing teams to hardcode authentication, routing, and monitoring logic directly into their AI applications, agentgateway acts as a drop-in AI-native data plane that deeply understands both the Model Context Protocol (MCP) and Agent-to-Agent (A2A) traffic. Through its cloud-native approach leveraging the Kubernetes Gateway API and xDS for dynamic configuration, it allows platform teams to quickly onboard agentic workflows alongside existing application micro services on multi-tenant platforms, making it easy to bridge to production without refactoring core agent or tool code. "

Vincent Caldeira

Chief Technology Officer, APAC at Red Hat

" Agentgateway is valuable because it treats AI workloads as a first-class part of the platform, not as something that needs its own parallel infrastructure. By unifying traffic management, policy, security, and observability across both AI and traditional services, it helps create a more coherent operating model for modern systems. That convergence reduces fragmentation, strengthens control, and gives platform teams a foundation that can scale with the next generation of workloads. "

Kevin Cao

Contributor of agentgateway and DevOps Engineer, Independent

" As AI agents and LLMs become essential components of modern infrastructure, navigating this landscape requires far more than just connecting to models — and that's exactly where agentgateway shines. Through my experience, I've seen how powerfully it handles LLM consumption, alongside critical architectural needs like robust authorization, seamless MCP integration, and secure agent-to-agent (A2A) communication. It truly is a game-changer for building reliable AI ecosystems. Here's to the amazing community and many more milestones ahead! "

Mehmet Hilmi Emel

AI & MLOps Engineer, ACEDEMAND IT Consulting Services

" We're really excited about agentgateway, as it brings a much-needed governance and security layer to agentic systems. We're using it as the central control plane to route and enforce policies on MCP traffic. It integrates really well with our MCP-G solution, giving us both enforcement and visibility. And it's been great to work with agentgateway especially for implementing JWT authentication, RBAC, and fine-grained control over tool access. "

Huzefa Hamdard

Senior Site Reliability Engineer, NIQ

Agentgateway travels around the world

Open Source Summit Europe 2025Ram Vennam gave a keynote at Open Source Summit Europe in Amsterdam, announcing the agentgateway donation to the Linux Foundation with Jim Zemlin. AI_dev Europe 2025Lin Sun gave a keynote at AI_dev in Amsterdam with a live demo of agentgateway running as part of Istio service mesh, enabling security, observability and traffic control for AI agents and MCP servers. KubeCon and CloudNativeCon NA 2025John Howard shared lessons learned building the next gen AI gateway, including the technical reasons why we created agentgateway from the ground up. Boston Kubernetes Meetup Nov 2025Nina Polshakova gave a talk including a live demo around kagent and agentgateway, keeping everyone engaged throughout the session. Open Source Summit South Korea 2025Mehmet Hilmi Emel gave his very first international talk on building custom MCP servers with FastMCP and integrating AI agents using Google ADK and agentgateway — so popular he gave it twice! API Days Paris 2025Lin Sun spoke about securing MCP servers with agentgateway at apidays Paris. Mobile World Congress Barcelona 2026Agentgateway traveled to Mobile World Congress Barcelona with two talks — one in the iconic talent arena, the other in the open gateway summit. MCPConference London 2026Duncan Doyle gave a talk at MCPConference London in February about securing MCP servers with agentgateway. MCPConference NY 2026Agentgateway traveled to the MCPConference in NY, showing the value of securing MCP servers with agentgateway made simpler.

Continuous technical innovation

Since Day 1, agentgateway keeps innovating: from the best performing AI gateway, to simplified configuration for LLM and MCP, to the ability to release faster, and more. Agentgateway also becomes an experimental dataplane for my favorite service mesh (also the most popular one), Istio. To learn more about the latest innovation in agentgateway, check out our recent 1.0 release blog.

Learn more about agentgateway

If you are in Amsterdam this week for KubeCon + CloudNativeCon Europe, join us at the Solo.io booth for the agentgateway birthday celebration on March 25th.

We welcome anyone passionate about the future of AI agents to join our community. Since launching community meetings in July, we’ve seen early participation and contributions from leading organizations including AWS, Microsoft, Red Hat, IBM, Zayo, Shell, Cisco, and Huawei.

To get involved:

Visit the agentgateway project GitHub.
Join the conversation on Discord.
Attend our weekly community meetings.

Let’s secure and govern the future of AI agents together.

Inference Serving with Superpowers

Mon, 23 Mar 2026 00:00:00 +0000

AI models and the surrounding ecosystem feel like they are evolving at the speed of light. A year ago, the question was how to expose model endpoints. Now the question is how to run inference as a real platform workload with policy, multi-tenancy, intelligent routing, cache locality, predictable latency, and room to scale across nodes and data centers.

This shift is exactly why agentgateway, llm-d, and the Gateway API Inference Extension (GIE) belong together.

llm-d is pushing new frontiers in intelligent scheduling, prefill/decode disaggregation, tiered KV caching, and workload-aware autoscaling. Agentgateway has quickly developed into a production-ready high performance AI gateway for LLM, MCP, A2A, and Kubernetes-native inference traffic. GIE has become the standard contract between the gateway layer and inference-aware scheduling.

Put those pieces together and you get something more useful than a demo stack. You get a Kubernetes-native path to production inference serving.

Inference is evolving from serving model endpoints to inference systems

Across llm-d, GIE, and the broader open inference ecosystem, three shifts stand out.

Inference routing is becoming first-class infrastructure. The gateway is no longer just a pass-through. It needs to understand models, priorities, failure modes, and inference objectives.
Performance gains increasingly come from scheduling and cache locality. Low-level GPU optimizations still matter, but platform wins now come from prefix-cache-aware routing, prefill/decode disaggregation, and better utilization of expensive accelerators.
Standards matter. Platform teams want the freedom to combine a gateway, scheduler, model server, and autoscaling strategy without custom glue or vendor lock-in at any layer.

That is the bigger story behind this stack. The market is not just asking for faster inference. It’s asking for composable inference systems based on industry standards.

Where the superpowers come from

%%{init: {
"theme": "base",
"flowchart": {
"htmlLabels": false
},
"themeVariables": {
"background": "#030712",
"lineColor": "#7734be",
"fontFamily": "Arial, Helvetica, sans-serif"
}
}}%%
flowchart LR
A["Apps and agents"]:::entry --> B["agentgateway<br/>policy, ingress, Gateway API"]:::gateway
B --> C["GIE InferencePool<br/>standard inference routing contract"]:::contract
C --> D["llm-d inference scheduler<br/>KV-aware and load-aware endpoint picking"]:::scheduler
D --> E["vLLM and serving backends"]:::backend
D --> F["KV cache tiers<br/>GPU, CPU, SSD, remote storage"]:::cache
classDef entry fill:#ede9fe,stroke:#a78bfa,color:#2c1d46,stroke-width:2px;
classDef gateway fill:#ddd6fe,stroke:#7734be,color:#2c1d46,stroke-width:2px;
classDef contract fill:#f5f3ff,stroke:#7734be,color:#2c1d46,stroke-width:2px;
classDef scheduler fill:#c4b5fd,stroke:#7c3aed,color:#2c1d46,stroke-width:2px;
classDef backend fill:#f5f3ff,stroke:#8b5cf6,color:#2c1d46,stroke-width:2px;
classDef cache fill:#ede9fe,stroke:#7c3aed,color:#2c1d46,stroke-width:2px;
linkStyle default stroke:#7734be,stroke-width:2px;

Each layer does a different job.

Agentgateway gives you the production traffic layer. It is a high-performance, Rust-based AI gateway that supports both standalone and Kubernetes deployment modes, recently achieved GA, and is the first GIE v1.4.0 conformant gateway.
GIE gives you the shared language between the gateway and the inference scheduler. Its InferencePool API and extension protocol let gateways route inference traffic without hard-coding scheduler behavior into the gateway itself.
llm-d gives you the serving intelligence. Its architecture focuses on intelligent inference scheduling, prefill/decode disaggregation, wide expert parallelism, tiered KV prefix caching, and workload autoscaling.

The clean separation while being integrated through standard interfaces is the superpower. agentgateway does not need to reimplement llm-d’s scheduler logic. llm-d does not need to reinvent gateway functionality, security policy, or Kubernetes networking APIs. GIE is the thread that stitches them all together. That makes the stack easier to understand and evolve over time.

Why this combination matters right now

The llm-d inference scheduler makes this relationship explicit. Its Endpoint Picker extends GIE and adds llm-d-specific capabilities such as P/D disaggregation. At the same time, the llm-d architecture leans into the exact optimizations that operators care about most:

Prefix-cache-aware routing
Utilization-based load balancing
Predicted latency balancing
Disaggregated prefill and decode
Tiered KV caching across host and storage layers

That is not abstract roadmap material. It’s the current capabilities and future direction of the project. llm-d’s move into CNCF Sandbox sharpens that story for users. It means the project is being positioned with vendor-neutral governance, broader cross-vendor participation, and tighter alignment with upstream cloud-native interfaces instead of a single-vendor roadmap. For users, that translates into three practical benefits:

More confidence that the inference stack is being built as a portable, vendor-agnostic platform capability.
Better interoperability pressure across adjacent projects in the inference and CNCF ecosystems.
Stronger trust signals around legal, security, and IP hygiene for teams that want to standardize on llm-d in production.

On the gateway side, the integration is now easier to consume. llm-d now includes the latest support for agentgateway and GIE.

Why GIE is a big deal

GIE matters because it keeps turning inference routing into a portable, testable interface instead of a stack-specific trick.

The v1.4.0 release added standalone Endpoint Picker (EPP) support, split conformance into its own Go module, and included InferencePool, Helm, and gRPC improvements such as appProtocol, FailOpen, and ALPN h2. It also continued pushing flow control, body-based routing, predicted latency, and data layer internals forward.

That matters for one simple reason: conformance is what turns a nice architecture diagram into a real platform choice.

Try it out

If you want to see this stack running end-to-end, use the agentgateway + llm-d demo.

The demo gives you a reproducible way to try agentgateway, GIE, and llm-d without requiring GPUs, using the latest agentgateway and GIE releases plus llm-d with the newly merged integration support. It lets you send OpenAI-compatible requests through agentgateway, compare gateway traffic against a direct baseline, and inspect the system in Prometheus and Grafana. Running the demo is as simple as:

git clone https://github.com/solo-io/agentgateway-llm-d.git
cd agentgateway-llm-d
cp config/demo.env.example config/demo.env
./scripts/demo.sh setup
./scripts/demo.sh port-forward start
./scripts/demo.sh e2e

After that, run ./scripts/demo.sh traffic start to light up the dashboards, or ./scripts/demo.sh walkthrough for a guided flow.

If you are building an inference platform on Kubernetes, agentgateway + llm-d + GIE is the stack worth paying attention to.

A Year In: agentgateway Hits v1.0.0 — and the Pieces Are Converging

Mon, 16 Mar 2026 00:00:00 +0000

A lot of open-source projects feel “promising” for a long time. And then, suddenly, a few signals land at once and you realize: it’s been a year, you’ve crossed 1 million image pulls, you’re approaching 2K GitHub stars… and v1.0.0 is here.

For agentgateway, that convergence looks like this:

~1 year of project velocity — the repo was created in March 2025
Crossing into the v1.0.0 release line
~2K stars on GitHub — nearing the threshold where adoption starts to accelerate
~1M image pulls — the usage curve is real

None of those numbers alone tell the story. All of them arriving together do.

What Is agentgateway?

Agentgateway is an open-source gateway and proxy hosted under the Linux Foundation. It routes HTTP and gRPC with the capabilities you expect from a mature data plane, and layers on first-class support for LLM inference, MCP tool servers, and A2A agent-to-agent communication—so teams can unify “normal” API traffic and agentic workloads instead of running parallel gateways.

It is built for the traffic patterns that traditional API gateways struggle with once MCP and A2A enter the picture, and it focuses on the gaps that show up the moment you run those workloads in production: governance, observability, multi-tenancy, and protocol-aware routing.

A few things that define the project:

Enterprise-grade security and multi-tenancy — built for shared infrastructure, not just single-tenant demos
Deep observability — including native OpenTelemetry support
Run anywhere — standalone binary or Kubernetes, same gateway either way
Performance and reliability first — designed to be the most mature LLM/MCP gateway available

Why v1.0 Matters

v1.0 marks an important milestone for the agentgateway project: it is now fully independent!

Decoupling from kgateway

The agentgateway Kubernetes controller is now included directly as part of the agentgateway project. Previously, the Kubernetes deployment was delivered through kgateway and followed its versioning and release lifecycle. With this change, agentgateway can now evolve and release independently.

We’d like to thank all the kgateway contributors who helped build and support this project along the way.

As part of this transition, the release version pattern has been updated to align with the versioning used by the agentgateway standalone binary. Going forward, both the Kubernetes-based agentgateway and the standalone binary use the same version numbers, simplifying releases and artifacts.

Documentation has also been updated to reflect the new structure, with v1.0.0 becoming the primary documentation version.

One release, one set of artifacts

The v1.0.0 alpha releases publish a consistent set of artifacts:

Docker images (controller + gateway)
Helm charts (agentgateway + agentgateway-crds)
- Binaries
  - cr.agentgateway.dev/agentgateway:v1.0.0
  - cr.agentgateway.dev/controller:v1.0.0
  - cr.agentgateway.dev/charts/agentgateway:v1.0.0
  - cr.agentgateway.dev/charts/agentgateway-crds:v1.0.0

🌟 New features

Agentgateway added a number of exciting features in v1.0. You can find the full list of new features in the release notes.

Kubernetes Gateway API version 1.5.0

The Kubernetes Gateway API dependency is updated to support version 1.5.0. This version introduces several changes, including:

XListenerSets promoted to ListenerSets: The experimental XListenerSet API is promoted to the standard ListenerSet API in version 1.5.0. The experimental XListenerSet API is promoted to ListenerSet in version 1.5.0. You no longer need the X prefix—existing experimental resources can continue to use v1alpha, but the API kind should be updated to ListenerSet.
AllowInsecureFallback mode for mTLS listeners: If you set up mTLS listeners on your agentgateway proxy, you can now configure the proxy to establish a TLS connection, even if the client TLS certificate could not be validated successfully. For more information, see the mTLS listener docs.
CORS wildcard support: The allowOrigins field now supports wildcard * origins to allow any origin. For an example, see the CORS guide.
BackendTLS: You can now apply BackendTLSPolicy resources to your routes to originate a TLS connection to a backend. For an example, see the BackendTLS guide.
Agentgateway continues to pass all Gateway API conformance tests (standard, extended, and experimental). TLSRoute is promoted; for users on the standard channel, use v1 instead of v1alpha2.

Simplified LLM configuration

The simplified llm configuration is designed specifically for LLM use cases. Use this approach when your primary goal is to route traffic to LLM providers. The docs are updated to use this simplified configuration. You can still use the previous route-based approach for backwards compatibility or for non-LLM use cases.

The simplified LLM configuration makes it easier when:

You are building an LLM gateway or AI proxy.
You need to route requests to one or more LLM providers.
You want model-based routing with header matching.
You need LLM-specific policies like JWT authentication or authorization.

The benefits of this approach are:

Concise: Less configuration needed for common LLM scenarios.
Model-centric: Focus on LLM models rather than HTTP routing.
LLM policies: Built-in support for JWT auth and authorization rules.
Easy multi-provider: Simple syntax for routing to multiple providers.

Example configuration:

# yaml-language-server: $schema=https://agentgateway.dev/schema/config

llm:
 models:
 - name: "*"
 provider: openAI
 params:
 apiKey: "$OPENAI_API_KEY"

You can even do more advanced policies and traffic matching:

# yaml-language-server: $schema=https://agentgateway.dev/schema/config

llm:
 policies:
 jwtAuth:
 issuer: agentgateway.dev
 audiences: [api.example.com]
 jwks:
 file: ./public-key.json
 authorization:
 rules:
 - 'jwt.email.endsWith("@example.com")'
 models:
 - name: claude-haiku
 provider: anthropic
 params:
 model: claude-3-5-haiku-20241022
 apiKey: "$ANTHROPIC_API_KEY"
 matches:
 - headers:
 - name: x-org
 value:
 exact: engineering
 - name: gpt-4
 provider: openAI
 params:
 model: gpt-4o
 apiKey: "$OPENAI_API_KEY"

CEL 2.0

This release includes a major refactor to the CEL implementation in agentgateway to improve scalability and performance. The following user facing changes were introduced:

The base64 function has been updated for compatibility with CEL-Go. Other functions retain previous names and behavior. Now, function names use dot notations, such as base64.encode. The old camel case names remain in place for backwards compatibility.
New string functions: The following string manipulation functions were added to the CEL library: startsWith, endsWith, stripPrefix, and stripSuffix. These functions align with the Google CEL-Go strings extension.
Null values fail: If a top-level variable returns a null value, the CEL expression now fails. Previously, null values always returned true. For example, the has(jwt) expression was previously successful if the JWT was missing or could not be found. Now, this expression fails.
Logical operators: Logical || and && operators now handle evaluation errors gracefully instead of propagating them. For example, a || b returns true if a is true even if b errors. Previously, the CEL expression failed.
Individual CEL expressions are now 5–500× faster, which has led to 50%+ end-to-end proxy performance improvements in some tests. Learn more in John Howard’s blog.

Make sure to update and verify any existing CEL expressions that you use in your environment.

For more information, see the CEL expression reference.

PreRouting phase support for policies

Agentgateway now supports PreRouting phase policies, allowing authentication and authorization checks to influence routing decisions before a route is selected. This makes it possible, for example, to route traffic to different LLM models or services based on request body content, headers, or JWT claims—all natively in the proxy, without external processors.

For example, you might want to route requests based on the model specified in the request body:

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
name: bbr
spec:
traffic:
phase: PreRouting
transformation:
request:
set:
- name: X-Gateway-Model-Name
value: 'json(request.body).model'

This native approach simplifies operations and dramatically improves performance compared to external processors: in tests, throughput increased 4.5x and p50 latency dropped by ~4.5× compared to an Istio-based external processor solution.

Availability

Agentgateway v1.0 is available for download on GitHub.

To get started with agentgateway, check out our getting started guide for standalone or Kubernetes.

Get Involved

The simplest way to get involved with kgateway is by joining our Discord and community meetings.

Contributors

Thanks to the 28 contributors who worked on agentgateway between the previous v0.12 release and v1.0, as well as the 115 contributors who have been with us since we began this journey!

Closing

The reason agentgateway feels like it’s hitting escape velocity isn’t any single milestone — it’s that all of the maturity signals are landing together. A year of velocity. A v1 release line. Real adoption numbers. And a packaging story that finally matches the ambition of the project.

Multi-Agent OpenClaw Architecture with a Kill Switch: Why Every AI Agent Needs a Gateway

Sat, 21 Feb 2026 00:00:00 +0000

The Setup

I run a multi-agent system. One coordinator agent handles user interaction, memory, and routing. Specialist sub-agents get spawned on demand for domain-specific tasks — security audits, network diagnostics, cloud management, infrastructure automation. Each specialist has its own system prompt, its own toolset, and runs on a different model.

It works. The specialists are good at their jobs. The coordinator knows when to delegate and when to handle things itself.

But here’s what keeps me up at night: what happens when one of these agents goes rogue?

A security agent with access to nmap and trivy decides to scan every host on the network in a loop. A cloud agent burns through $500 of Opus tokens chasing a hallucinated Terraform state or decides to reconfigure your Istio ambient mesh routing because it misread a waypoint proxy status. A general agent with SSH access starts “fixing” things on production hosts that don’t need fixing.

Without a control plane between your agents and the outside world, you have no way to stop any of this. No kill switch. No cost ceiling. No audit trail. No rate limits. Just agents with direct access to LLMs and tools, hoping nothing goes wrong.

That’s not engineering. That’s negligence.

The Architecture

Here’s what I actually run. Every LLM call and every MCP tool invocation from every agent — coordinator and specialists alike — routes through agentgateway.

┌─────────────────────────────────────────────────┐
│ User (Seb) │
│ Telegram / Discord / CLI │
└────────────────────┬────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ Coordinator Agent │
│ (Jacob) │
│ │
│ • User interaction & conversation │
│ • Memory management (MEMORY.md) │
│ • Task triage & routing │
│ • Context assembly for specialists │
│ • Result synthesis & delivery │
└──┬──────────┬──────────┬──────────┬─────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────┐ ┌──────┐ ┌──────┐ ┌──────────┐
│ Sec │ │ Net │ │Cloud │ │ General │
│Agent │ │Agent │ │Agent │ │ Agent │
└──┬───┘ └──┬───┘ └──┬───┘ └────┬─────┘
│ │ │ │
└─────────┴─────────┴────────────┘
│
│ ALL traffic
▼
┌──────────────────────────────────────────┐
│ Kubernetes Cluster │
│ ┌──────────────────────────────┐ │
│ │ agentgateway │ │
│ │ (Pod) │ │
│ │ │ │
│ │ • Kill switch │ │
│ │ • Rate limiting │ │
│ │ • Cost controls │ │
│ │ • JWT auth + RBAC │ │
│ │ • Observability (OTel) │ │
│ │ • Tool poisoning protection │ │
│ └──────┬───────────┬───────────┘ │
│ │ │ │
└─────────┼───────────┼────────────────────┘
│ │
┌────▼────┐ ┌───▼────────────────┐
│ LLMs │ │ MCP Servers │
│Anthropic│ │ nmap, trivy │
│ OpenAI │ │ aws-cli, kubectl │
│ xAI │ │ istioctl, docker │
└─────────┘ └───────────────────-┘

Nothing reaches an LLM or a tool without passing through the gateway. That’s the entire point.

The Agents

Coordinator: Jacob

The coordinator is the only agent that talks to the user. It owns the conversation, manages memory (a persistent MEMORY.md that carries context across sessions), and decides which specialist to invoke for each request.

When a task comes in, the coordinator classifies it and builds a context payload — the relevant portion of memory, the specific question, any constraints — and spawns a specialist. The specialist does its work, returns a result, and dies. Stateless. Disposable.

The coordinator synthesizes the result and delivers it back to the user. If a task spans multiple domains, the coordinator fans out to multiple specialists in parallel.

Model: Sonnet — fast enough for routing, smart enough for context assembly.

Security Agent

Domain: Vulnerability scanning, CVE analysis, firewall rules, IAM audits, compliance checks
Tools: nmap, trivy, falco, OWASP ZAP, CIS benchmarks, secrets scanning
Model: Opus — high reasoning for threat analysis
Access: Read-only on infra by default, escalation required for remediation

Network Agent

Domain: DNS, routing, load balancing, VPN, firewall config, traffic analysis
Tools: dig, traceroute, tcpdump, iperf3, netstat, ip, iptables, tshark
Model: Sonnet — fast, good for diagnostic tasks
Access: Network interfaces, DNS servers, routing tables

Cloud Agent

Domain: AWS/GCP/Azure resource management, Terraform, cost optimization, architecture, Kubernetes, Istio service mesh, ambient mesh
Tools: aws-cli, gcloud, az, terraform, kubectl, helm, istioctl
Model: Sonnet — balance of speed and capability
Access: Cloud provider credentials (scoped IAM roles), Kubernetes clusters, Istio control plane

General / Infra Agent

Domain: Proxmox, Docker, Linux admin, Git, CI/CD, general automation
Tools: ssh, docker, git, systemctl, proxmox API, cron
Model: Sonnet (routine ops) or Haiku (simple tasks)
Access: Full local system, Proxmox API, SSH to hosts

Memory System

OpenClaw Memory System Persistent recall across sessions using markdown files and vector search.

Two memory layers — Daily logs (memory/YYYY-MM-DD.md) for running notes, and MEMORY.md for curated long-term facts and preferences.
Disk-based memory — No RAM between sessions, so everything must be written to disk to be remembered.
Vector embeddings — Files are chunked and converted to vectors using Qwen3-Embedding-0.6B via node-llama-cpp, fully local and private.
Hybrid search — Combines vector similarity (semantic) with BM25 full-text search (keyword) for accurate recall.
Temporal decay — Recent memories are weighted higher in search results.
memory_search / memory_get — Agent tools to find relevant past context or read a file directly.
Auto-compaction — When token limits are hit, the agent is prompted to save important context before compacting so nothing is lost.

Routing Logic

The coordinator classifies each request and routes to the appropriate specialist:

Keywords	Routes To
CVE, vulnerability, audit, compliance, secrets	Security Agent
DNS, firewall, routing, VPN, latency, ports	Network Agent
AWS, Terraform, GCP, Azure, S3, EC2, cost, Istio, mesh, Kubernetes, k8s	Cloud Agent
VM, Docker, git, systemd, Proxmox, backup	General Agent

Ambiguous requests stay with the coordinator. Multi-domain tasks fan out to multiple specialists in parallel.

Why Every Agent Goes Through the Gateway

This is the part that matters. Here’s why I don’t let any agent — not even the coordinator — talk to LLMs or tools directly.

The Doom Scenario

Picture this: your cloud agent is debugging a Terraform plan. It calls Opus to reason about a complex state migration. The model hallucinates a resource dependency. The agent re-plans, calls the model again for clarification, gets another hallucination, retries with more context (bigger prompt, more tokens), and enters a loop. Each iteration costs more than the last because the context window keeps growing.

Without a gateway: you find out when the invoice arrives. $2,000 spent on a conversation with itself.

With agentgateway: the agent hits a token-per-minute ceiling after the third iteration. The request is rejected. You get an alert. You investigate. Total damage: $12.

That’s not a hypothetical. That’s Tuesday.

Kill Switch

agentgateway gives me a single point where I can shut everything down. If I see an agent misbehaving — through the metrics, through the traces, through an alert — I can:

Revoke the JWT for that specific agent’s identity. Immediate. That agent can’t make another LLM call or tool invocation.
Update the rate limit to zero for that agent class. Every security agent stops. Every cloud agent stops. Surgical.
Pull the gateway entirely. Nuclear option. Everything stops. Nothing reaches any LLM or tool.

Without a gateway, killing a rogue agent means finding the pod, kubectl exec-ing into the right node, and hoping you’re faster than the agent. With a gateway running in Kubernetes, it’s a config change — or a kubectl rollout restart away from a full reset.

Cost Controls

Every agent has a budget. Not a suggestion — a hard limit enforced at the gateway level.

# Security agent route — Opus workloads
policies:
 localRateLimit:
 - maxTokens: 50000
 tokensPerFill: 50000
 fillInterval: 1m
 type: tokens
 - maxTokens: 20
 tokensPerFill: 20
 fillInterval: 1m
 type: requests

# Cloud agent route — higher throughput
policies:
 localRateLimit:
 - maxTokens: 100000
 tokensPerFill: 100000
 fillInterval: 1m
 type: tokens
 - maxTokens: 30
 tokensPerFill: 30
 fillInterval: 1m
 type: requests

# General agent route — simple ops
policies:
 localRateLimit:
 - maxTokens: 20000
 tokensPerFill: 20000
 fillInterval: 1m
 type: tokens
 - maxTokens: 15
 tokensPerFill: 15
 fillInterval: 1m
 type: requests

Each route gets a token-bucket rate limit scoped by the route’s identity. The security agent running Opus gets 50k tokens per minute. That’s enough for serious threat analysis but not enough to bankrupt me on a hallucination loop. The general agent on Haiku gets 20k — simple ops don’t need more.

agentgateway tracks token usage per provider and per model with agentgateway_gen_ai_client_token_usage metrics, tagged with provider, model, and operation labels. I know exactly what each agent costs, in real time.

Rate Limiting

Rate limits aren’t just about cost. They’re about preventing an agent from overwhelming a downstream system.

A network agent running nmap scans through an MCP tool server could, in theory, scan your entire /16 network if nobody stops it. Rate limiting at the gateway means the agent gets N tool calls per minute, period. It can’t outrun the limit no matter how convinced it is that it needs to scan “just one more subnet.”

Same for LLM calls. An agent that retries on every 429 or timeout — something LLM providers actually rate-limit you for — gets its retries throttled at the gateway before the provider even sees them.

Governance and RBAC

Each agent has a JWT identity with scoped permissions. The security agent can call nmap and trivy tools but cannot call terraform apply. The cloud agent can call terraform plan but not ssh. The general agent can SSH to designated hosts but cannot touch cloud credentials.

This is enforced at the gateway with CEL expressions in mcpAuthorization rules:

# Security agent backend
mcpAuthorization:
 rules:
 - >-
 jwt.agent_role == "security" && (
 mcp.tool.name.startsWith("nmap") ||
 mcp.tool.name.startsWith("trivy") ||
 mcp.tool.name.startsWith("falco")
 )

# Cloud agent backend
mcpAuthorization:
 rules:
 - >-
 jwt.agent_role == "cloud" && (
 mcp.tool.name.startsWith("terraform") ||
 mcp.tool.name.startsWith("kubectl") ||
 mcp.tool.name.startsWith("istioctl") ||
 mcp.tool.name.startsWith("helm")
 )

Even if a specialist agent’s system prompt gets jailbroken and it tries to invoke tools outside its domain, the gateway blocks it. If a tool isn’t matched by a rule, it’s automatically filtered from the tools/list response — the agent literally cannot see tools it doesn’t have access to.

And since unmatched tools are denied by default, I only whitelist what’s explicitly allowed. Any tool not covered by an mcpAuthorization rule is invisible to the agent. For HTTP-level operations, I add explicit deny rules:

authorization:
 rules:
 - deny: 'request.path.contains("delete")'
 - deny: 'request.path.contains("destroy")'
 - deny: 'request.path.contains("drop")'

No agent gets to run destructive operations without explicit human escalation. Period.

Full Observability

Every LLM call and every tool invocation generates OpenTelemetry traces. Every trace is tagged with the agent identity that triggered it.

I can see:

Which agent made the call
What prompt was sent to the LLM
What tool was invoked with what arguments
How many tokens were consumed
How long it took
Whether it succeeded or failed

┌─ Trace: security-agent-cve-scan ──────────────────┐
│ │
│ initialize 12ms mcp-session-setup │
│ list_tools 8ms tool-discovery │
│ call_tool(nmap) 4.2s scan-target-host │
│ llm_call(opus) 3.1s analyze-scan-results │
│ call_tool(trivy) 6.8s container-vuln-scan │
│ llm_call(opus) 2.4s synthesize-findings │
│ │
│ Total: 16.5s | Tokens: 12,847 | Cost: $0.38 │
└─────────────────────────────────────────────────────┘

Metrics go to Prometheus. Traces go to Jaeger. LLM-specific telemetry goes to Langfuse for prompt/completion pair analysis. All of it through agentgateway’s built-in OpenTelemetry support — no instrumentation code in the agents themselves.

When something goes wrong, I don’t grep through logs hoping to find what happened. I open a dashboard and see exactly which agent, which call, which tool, at what time, with what parameters.

Design Decisions

Specialists are stateless, spawned per task. Simple and cost-effective. No long-running agent processes consuming resources while idle. The coordinator is the only persistent component.

Coordinator owns all memory. Specialists get context injected per request. They don’t need to remember previous conversations — the coordinator handles continuity.

Model per agent. Opus for security (high-stakes reasoning). Sonnet for network/cloud (speed + capability balance). Haiku for simple ops (cost efficiency). Each agent gets the cheapest model that’s good enough for its domain.

Tool isolation. Each specialist only gets the tools it needs. Not through prompt instructions (which can be jailbroken) but through gateway-enforced RBAC (which can’t).

Single gateway for all traffic. Not one gateway per agent. Not a sidecar pattern. One agentgateway instance running in Kubernetes that every agent routes through. One place to set policy, one place to monitor, one place to kill. K8s gives me rolling updates, health checks, and resource limits on the gateway itself — so the control plane has its own control plane.

Extensible. New domain = new agent config + system prompt + tool set. The coordinator’s routing logic gets a new keyword match. The gateway gets a new JWT scope. No architectural changes needed.

Running agentgateway in Kubernetes

agentgateway runs as a deployment in my Kubernetes cluster. This isn’t just convenience — it’s operational discipline. The gateway that controls all my agents is itself managed by K8s primitives: health checks, resource limits, rolling updates, and restart policies.

The kill switch becomes even simpler in K8s. Scale to zero replicas and every agent loses its gateway instantly:

kubectl scale deployment agentgateway -n agent-infra --replicas=0

Everything stops. Scale back up when you’ve fixed the issue. The gateway comes back with the same config, same policies, same state.

The cloud agent — the one that handles Kubernetes, Istio, and ambient mesh — is particularly interesting in this setup. It manages the same cluster that hosts the gateway. That’s a circular dependency I’ve thought carefully about: the agent that manages K8s infrastructure talks through a gateway that runs on K8s infrastructure. The circuit breaker here is the RBAC policy — the cloud agent’s JWT scope explicitly excludes the agent-infra namespace. It can manage workloads, configure Istio routing, and deploy ambient mesh policies, but it cannot touch the gateway deployment itself.

Why agentgateway and Not a Traditional Proxy

Traditional API gateways (Envoy, Kong, NGINX) were built for HTTP request/response. AI agent traffic is fundamentally different:

MCP is stateful. Agents maintain long-lived sessions with tool servers. Requests and responses are tied to session context. Traditional gateways don’t maintain session awareness.
LLM calls are long-running. A single inference call can take 30+ seconds with streaming. Connection timeouts designed for web APIs don’t apply.
Token-based economics. Cost isn’t about request count — it’s about token count. A gateway that can’t count tokens can’t enforce budgets.
Bidirectional communication. MCP servers can push messages back to clients asynchronously. This breaks the request/response model traditional gateways assume.

agentgateway is purpose-built for this. Written in Rust for performance and memory safety on stateful, long-lived connections. Understands MCP sessions natively. Counts tokens per-provider. Handles fan-out patterns where one agent call becomes multiple downstream requests.

It’s open source, Apache 2.0 licensed, and part of the Linux Foundation. No vendor lock-in.

The Takeaway

A multi-agent system without a control plane is a liability. Every agent you deploy is a potential cost bomb, a potential security breach, a potential “I can’t believe nobody caught that” incident.

The architecture is straightforward:

One coordinator that handles users and routes tasks
Specialist agents that are stateless, scoped, and disposable
One gateway that sees everything, controls everything, and logs everything

The coordinator decides what gets done. The gateway decides whether it’s allowed to happen. That separation is what makes the system safe to run autonomously.

agentgateway isn’t optional in this architecture. It’s the thing that makes the entire system possible without me staring at a terminal 24/7 wondering if an agent is about to do something catastrophic.

Build the agents. Put the gateway in front. Sleep at night.

Resources:

MCP Multiplexing with AgentGateway

Fri, 20 Feb 2026 00:00:00 +0000

Introduction

As your agentic AI environment grows, you end up with a sprawl of MCP servers — one for time utilities, one for general-purpose tools, one for Slack, another for GitHub. Each server exposes different tools, runs on a different port, and needs its own connection configuration.

Your MCP clients (Cursor, Claude Desktop, VS Code) shouldn’t need to know about every server individually. They should connect to one endpoint and see all available tools.

This guide walks you through MCP Multiplexing — federating multiple MCP servers behind a single AgentGateway endpoint.

We’ll deploy two MCP servers (mcp-server-everything for utility tools and mcp-website-fetcher for fetching web content), multiplex them behind AgentGateway, and connect from any IDE.

What You Get

One endpoint, many servers: Clients connect to /mcp and see tools from all federated servers
Automatic tool namespacing: Tools are prefixed with the server name to avoid conflicts
Label-based federation: Add servers by just adding a Kubernetes label — no config changes
IDE-agnostic: Works with Cursor, VS Code, Claude Code, Windsurf, OpenCode

Architecture

Prerequisites

Docker installed and running
kind installed
kubectl and helm installed
At least one MCP client: Cursor, VS Code, Claude Code, Windsurf, or OpenCode

Step 1: Create a Kind Cluster

kind create cluster --name agentgateway-mcp

Verify it’s running:

kubectl get nodes

Step 2: Install AgentGateway OSS

Deploy the Kubernetes Gateway API CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Install the AgentGateway CRDs and control plane:

helm upgrade -i --create-namespace \
 --namespace agentgateway-system \
 --version v1.3.11.4.0-alpha.1 \
 agentgateway-crds oci://ghcr.io/kgateway-dev/charts/agentgateway-crds

helm upgrade -i -n agentgateway-system \
 agentgateway oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --version v1.3.11.4.0-alpha.1

Verify the control plane is running:

kubectl get pods -n agentgateway-system

You should see the agentgateway control plane pod in Running state.

Step 3: Create an AgentGateway Proxy

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: mcp-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: http
 port: 80
 protocol: HTTP
EOF

Wait for the proxy pod:

kubectl rollout status deploy/mcp-gateway -n agentgateway-system --timeout=60s

Step 4: Deploy Two MCP Servers

Server 1: mcp-server-everything

This is the reference MCP test server from the Model Context Protocol project. It provides utility tools like echo, add, longRunningOperation, and more. The streamableHttp argument tells it to listen over HTTP instead of stdio — which is exactly the transport AgentGateway’s multiplex feature requires.

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mcp-server-everything
 namespace: agentgateway-system
 labels:
 app: mcp-server-everything
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mcp-server-everything
 template:
 metadata:
 labels:
 app: mcp-server-everything
 spec:
 containers:
 - name: mcp-server-everything
 image: node:20-alpine
 command: ["npx"]
 args: ["-y", "@modelcontextprotocol/server-everything", "streamableHttp"]
 ports:
 - containerPort: 3001
---
apiVersion: v1
kind: Service
metadata:
 name: mcp-server-everything
 namespace: agentgateway-system
 labels:
 app: mcp-server-everything
 mcp-federation: "true"
spec:
 selector:
 app: mcp-server-everything
 ports:
 - protocol: TCP
 port: 3001
 targetPort: 3001
 appProtocol: agentgateway.dev/mcp
 type: ClusterIP
EOF

Server 2: mcp-website-fetcher

This server provides a fetch tool that can retrieve and extract content from any URL — useful for giving your AI assistant web browsing capabilities. It’s already HTTP-native, no wrapper needed.

kubectl apply -f- <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
 name: mcp-website-fetcher-fix
 namespace: agentgateway-system
data:
 app.py: |
 import httpx
 from mcp.server.fastmcp import FastMCP
 mcp = FastMCP(
 "mcp-website-fetcher",
 host="0.0.0.0",
 port=8000,
 streamable_http_path="/mcp",
 stateless_http=True,
 )
 @mcp.tool()
 async def fetch(url: str) -> str:
 """Fetches a website and returns its content"""
 headers = {"User-Agent": "MCP Test Server"}
 async with httpx.AsyncClient(follow_redirects=True, headers=headers) as client:
 response = await client.get(url)
 response.raise_for_status()
 return response.text
 if __name__ == "__main__":
 mcp.run(transport="streamable-http")
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mcp-website-fetcher
 namespace: agentgateway-system
spec:
 selector:
 matchLabels:
 app: mcp-website-fetcher
 template:
 metadata:
 labels:
 app: mcp-website-fetcher
 spec:
 containers:
 - name: mcp-website-fetcher
 image: ghcr.io/peterj/mcp-website-fetcher:main
 imagePullPolicy: Always
 command: ["python3", "/app/app.py"]
 volumeMounts:
 - name: fix
 mountPath: /app/app.py
 subPath: app.py
 volumes:
 - name: fix
 configMap:
 name: mcp-website-fetcher-fix
---
apiVersion: v1
kind: Service
metadata:
 name: mcp-website-fetcher
 namespace: agentgateway-system
 labels:
 app: mcp-website-fetcher
 mcp-federation: "true"
spec:
 selector:
 app: mcp-website-fetcher
 ports:
 - port: 80
 targetPort: 8000
 appProtocol: agentgateway.dev/mcp
 type: ClusterIP
EOF

Wait for both servers to be ready:

kubectl rollout status deploy/mcp-server-everything -n agentgateway-system --timeout=60s
kubectl rollout status deploy/mcp-website-fetcher -n agentgateway-system --timeout=60s

Key detail: Both services have the label mcp-federation: "true" and appProtocol: agentgateway.dev/mcp. This is how AgentGateway discovers and federates them.

Step 5: Create the Multiplexed MCP Backend

Here’s where the magic happens. Instead of pointing to individual servers, you use a label selector to match all services with mcp-federation: "true":

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: mcp-federated
 namespace: agentgateway-system
spec:
 mcp:
 targets:
 - name: mcp-servers
 selector:
 services:
 matchLabels:
 mcp-federation: "true"
EOF

This single backend automatically discovers both mcp-server-everything and mcp-website-fetcher — and any future MCP server you deploy with the mcp-federation: "true" label.

Step 6: Create the HTTPRoute

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: mcp-route
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: mcp-gateway
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /mcp
 backendRefs:
 - name: mcp-federated
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Step 7: Port-Forward and Test

Expose the gateway locally:

kubectl port-forward -n agentgateway-system deployment/mcp-gateway 8080:80

Test with MCP Inspector

npx @modelcontextprotocol/inspector@0.21.2

Connect with:

Transport: Streamable HTTP
URL: http://localhost:8080/mcp

Click Tools → List Tools. You should see tools from both servers, namespaced:

mcp-server-everything-3001_echo — echo a message
mcp-server-everything-3001_add — add two numbers
mcp-server-everything-3001_longRunningOperation — simulate a long task
mcp-website-fetcher-80_fetch — fetch and extract content from a URL

One endpoint. Two servers. All tools.

Step 8: Connect from Your IDE

With port-forward running, configure your IDE:

Cursor

Create or edit ~/.cursor/mcp.json:

{
 "mcpServers": {
 "federated-tools": {
 "url": "http://localhost:8080/mcp"
 }
 }
}

VS Code (with GitHub Copilot)

Add to settings.json:

{
 "github.copilot.chat.mcp.servers": {
 "federated-tools": {
 "url": "http://localhost:8080/mcp"
 }
 }
}

Claude Code

claude mcp add federated-tools --transport sse http://localhost:8080/mcp

Windsurf

Create or edit ~/.windsurf/mcp.json:

{
 "mcpServers": {
 "federated-tools": {
 "url": "http://localhost:8080/mcp"
 }
 }
}

OpenCode

Add to opencode.json:

{
 "mcp": {
 "servers": {
 "federated-tools": {
 "type": "sse",
 "url": "http://localhost:8080/mcp"
 }
 }
 }
}

Now try asking your IDE:

“Echo back: AgentGateway is awesome” — uses the everything server
“Add 42 and 58” — uses the everything server
“Fetch the content from https://agentgateway.dev” — uses the website fetcher

The IDE doesn’t know or care that these tools come from different servers. It’s all one MCP endpoint.

Adding More MCP Servers

The beauty of label-based federation: adding a new server is just a deployment + service with the mcp-federation: "true" label. No AgentGateway config changes needed — the backend’s label selector picks it up automatically.

Cleanup

kubectl delete httproute mcp-route -n agentgateway-system
kubectl delete agentgatewaybackend mcp-federated -n agentgateway-system
kubectl delete deploy mcp-server-everything mcp-website-fetcher -n agentgateway-system
kubectl delete svc mcp-server-everything mcp-website-fetcher -n agentgateway-system
kind delete cluster --name agentgateway-mcp

Conclusion

MCP multiplexing solves one of the biggest challenges in production agentic AI environments: server sprawl. Developers shouldn’t need to configure connections to every MCP server individually. One gateway endpoint, all tools.

Federate all your MCP servers behind AgentGateway, add new servers with a label, and every connected client sees the new tools immediately. No agent code changes. No client reconfiguration.

One gateway. All your tools.

Connect Any IDE to GitHub MCP Server Through Agentgateway

Thu, 19 Feb 2026 00:00:00 +0000

Introduction

Modern AI-powered IDEs and coding agents ship with built-in MCP client support. That means you can connect them to GitHub’s remote MCP server to give your AI assistant direct access to issues, pull requests, code search, and repository management.

But connecting directly to GitHub means no visibility into what tools are being called, no rate limiting, and no centralized credential management. Every developer has their own PAT, every tool call goes straight to GitHub ungoverned.

This guide shows you how to route MCP traffic from any IDE — Cursor, VS Code, Windsurf, Claude Code, or OpenCode — to GitHub’s remote MCP server through Solo AgentGateway. You deploy the gateway once, and every IDE on your team connects through it.

What You Get

One gateway, every IDE: Deploy AgentGateway once, connect all your tools
Centralized credentials: GitHub PAT lives in a Kubernetes secret, not on every developer laptop
Rate limiting: Control tool call volume per IDE or per user
Observability: OpenTelemetry traces for every MCP interaction
No self-hosted MCP server: GitHub hosts the MCP server at api.githubcopilot.com — you just proxy

Architecture

Prerequisites

Docker installed and running
kind installed
kubectl and helm installed
A GitHub Personal Access Token (PAT)
At least one of: Cursor, VS Code (with Copilot), Windsurf, Claude Code, or OpenCode

Step 1: Create a Kind Cluster

kind create cluster --name agentgateway

Verify it’s running:

kubectl get nodes

Step 2: Install AgentGateway

Deploy the Kubernetes Gateway API CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml

Install the AgentGateway CRDs and control plane:

helm upgrade -i --create-namespace \
 --namespace agentgateway-system \
 --version v1.3.11.4.0-alpha.1 agentgateway-crds oci://ghcr.io/kgateway-dev/charts/agentgateway-crds

helm upgrade -i -n agentgateway-system agentgateway oci://ghcr.io/kgateway-dev/charts/agentgateway \
--version v1.3.11.4.0-alpha.1

Verify the control plane is running:

kubectl get pods -n agentgateway-system

Step 3: Create an AgentGateway Proxy

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: http
 port: 80
 protocol: HTTP
EOF

Wait for the proxy pod to be ready:

kubectl rollout status deploy/agentgateway-proxy -n agentgateway-system

Step 4: Configure the GitHub Remote MCP Backend

GitHub hosts a remote MCP server at https://api.githubcopilot.com/mcp/. We point AgentGateway at it instead of deploying our own.

Create a secret with your GitHub PAT:

kubectl create secret generic github-mcp-secret \
 -n agentgateway-system \
 --from-literal=Authorization="Bearer ghp_your_token_here"

Create the backend:

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: github-mcp-backend
 namespace: agentgateway-system
spec:
 mcp:
 targets:
 - name: mcp-target
 static:
 host: api.githubcopilot.com
 port: 443
 path: /mcp/
 policies:
 auth:
 secretRef:
 name: github-mcp-secret
 tls:
 sni: api.githubcopilot.com
EOF

Key details:

path: /mcp/ — GitHub’s MCP endpoint
tls.sni — required for HTTPS on port 443
auth.secretRef — injects the Authorization: Bearer <PAT> header automatically

Step 5: Create the HTTPRoute

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: github-mcp
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /github
 backendRefs:
 - name: github-mcp-backend
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Step 6: Port-Forward

Expose the gateway locally:

kubectl port-forward -n agentgateway-system deployment/agentgateway-proxy 8080:80

Leave this running. All IDE configs below use http://localhost:8080.

IDE Configuration

Each IDE has its own MCP config format. Pick yours below — or configure all of them to use the same gateway endpoint.

Cursor

Create or edit ~/.cursor/mcp.json (global) or .cursor/mcp.json (per-project):

{
 "mcpServers": {
 "github": {
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

Restart Cursor → Settings → MCP → verify GitHub tools appear.

VS Code (with GitHub Copilot)

Add to your VS Code settings.json (Cmd/Ctrl + , → search “mcp”):

{
 "github.copilot.chat.mcp.servers": {
 "github": {
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

Or for workspace-specific config, add to .vscode/settings.json.

Reload VS Code (Cmd/Ctrl + Shift + P → “Developer: Reload Window”) → open Copilot Chat → tools should list GitHub MCP tools.

Windsurf

Create or edit ~/.windsurf/mcp.json (global) or .windsurf/mcp.json (per-project):

{
 "mcpServers": {
 "github": {
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

Restart Windsurf → verify connection in MCP settings.

Claude Code

Add via CLI:

claude mcp add github --transport sse http://localhost:8080/github/mcp

Or add to your project’s .mcp.json:

{
 "mcpServers": {
 "github": {
 "type": "sse",
 "url": "http://localhost:8080/github/mcp"
 }
 }
}

OpenCode

Add to your opencode.json config:

{
 "mcp": {
 "servers": {
 "github": {
 "type": "sse",
 "url": "http://localhost:8080/github/mcp"
 }
 }
 }
}

Step 7: Test It

Open your IDE’s AI chat and try:

“List open issues in my repo”
“Create a new branch called feature/test”
“Search for files referencing AgentGateway”

Every tool call flows through AgentGateway regardless of which IDE you’re using. The gateway handles auth, logs the interaction, and forwards to GitHub.

Verifying Traffic

Check proxy logs:

kubectl logs -n agentgateway-system deploy/agentgateway-proxy --tail=20

You should see MCP connection events and tool call forwards to api.githubcopilot.com.

IDE Comparison

Note: All IDEs connect to the same http://localhost:8080/github/mcp endpoint. The gateway doesn’t care which client is calling — it applies the same auth, rate limiting, and observability to all of them.

What’s Next

Add more MCP servers: Slack, Notion, Jira — all behind the same gateway, same endpoint pattern
Security policies: JWT auth, prompt guards, tool-level RBAC with AgentgatewayPolicy
Team rollout: Share the gateway endpoint with your team — one PAT, centrally managed, no credentials on laptops
Per-IDE rate limiting: Different limits for different tools or users

Cleanup

kubectl delete httproute github-mcp -n agentgateway-system
kubectl delete agentgatewaybackend github-mcp-backend -n agentgateway-system
kubectl delete secret github-mcp-secret -n agentgateway-system
kind delete cluster --name agentgateway

Conclusion

Every AI-powered IDE now speaks MCP. But connecting each one directly to backend servers means fragmented credentials, zero visibility, and no governance. AgentGateway gives you a single control point — deploy it once, connect every IDE on your team, and get auth, rate limiting, and observability on every tool call.

One gateway. Every IDE. Full visibility.

Open Source LLM Observability: Tracing AI Calls with AgentGateway and Langfuse

Wed, 18 Feb 2026 00:00:00 +0000

Introduction

You’ve got your AI gateway routing LLM traffic. But can you actually see what’s happening? Which models are being called, how many tokens are being consumed, what prompts are going in and what completions are coming back?

This guide shows you how to integrate Langfuse with Solo AgentGateway to get full observability over every LLM request — without touching your application code. We’ll go from zero to traced requests on a local kind cluster in under 10 minutes.

Why This Matters

AgentGateway already captures rich telemetry about every LLM request: model, tokens, latency, route, and security policy actions. By forwarding those traces to Langfuse, you get:

Full prompt and completion visibility across all LLM providers (OpenAI, Anthropic, xAI, etc.)
Token usage and cost tracking per model, route, and user
Latency analysis with gateway-level metadata — which route, which backend, which policy fired
Zero application changes — tracing happens at the gateway layer, not in your app

This is the power of observability at the infrastructure layer. Your developers don’t need to instrument anything. Every LLM call that flows through the gateway is automatically captured.

Architecture

Here’s what we’re building:

┌──────────────┐ ┌────────────────────────┐ ┌─────────────────┐
│ Your App / │ │ Solo AgentGateway │ │ LLM Provider │
│ AI Agent │────▶│ (Gateway API) │────▶│ (OpenAI, etc) │
└──────────────┘ └───────────┬────────────┘ └─────────────────┘
│
OTLP Traces (gRPC)
│
┌───────────▼────────────┐
│ OpenTelemetry │
│ Collector │
└─────┬───────────┬──────┘
│ │
OTLP HTTP OTLP gRPC
│ │
┌─────▼──┐ ┌─────▼──────────┐
│Langfuse│ │Other backends │
│ UI │ │(Jaeger, etc.) │
└────────┘ └─────────────────┘

AgentGateway natively emits OpenTelemetry traces for every LLM request. A lightweight OTel Collector receives those traces and forwards them to Langfuse via OTLP HTTP. The collector can also fan-out to additional backends like Jaeger, Datadog, or ClickHouse if you need traces in multiple places.

Prerequisites

Docker installed and running
kind installed
kubectl installed
helm installed
A Langfuse account (free tier works) or self-hosted instance
An OpenAI API key (or any supported LLM provider)

Step 1: Create a Kind Cluster

kind create cluster --name agentgateway
kubectl cluster-info --context kind-agentgateway

Step 2: Install the Gateway API CRDs

AgentGateway uses the standard Kubernetes Gateway API for configuration:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Step 3: Install AgentGateway

Install the CRDs and control plane:

helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/helm/agentgateway-crds \
 --version 2.1.0 \
 --namespace agentgateway-system \
 --create-namespace

helm upgrade -i agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system

Verify everything is running:

kubectl get pods -n agentgateway-system
kubectl get gatewayclass agentgateway

Step 4: Get Your Langfuse Credentials

Log in to Langfuse and go to Settings → API Keys. Create a new key pair (or use an existing one). Base64 encode them for the collector config:

echo -n "pk-lf-YOUR_PUBLIC_KEY:sk-lf-YOUR_SECRET_KEY" | base64

Step 5: Deploy the OTel Collector

The collector bridges AgentGateway (OTLP gRPC) and Langfuse (OTLP HTTP). Create langfuse-collector.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
 name: langfuse-otel-collector-config
 namespace: agentgateway-system
data:
 config.yaml: |
 receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
 exporters:
 otlphttp/langfuse:
 endpoint: https://cloud.langfuse.com/api/public/otel
 headers:
 Authorization: "Basic <YOUR_BASE64_CREDENTIALS>"
 retry_on_failure:
 enabled: true
 initial_interval: 5s
 max_interval: 30s
 max_elapsed_time: 300s
 processors:
 batch:
 send_batch_size: 1000
 timeout: 5s
 service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlphttp/langfuse]
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: langfuse-otel-collector
 namespace: agentgateway-system
 labels:
 app: langfuse-otel-collector
spec:
 replicas: 1
 selector:
 matchLabels:
 app: langfuse-otel-collector
 template:
 metadata:
 labels:
 app: langfuse-otel-collector
 spec:
 containers:
 - name: otel-collector
 image: otel/opentelemetry-collector-contrib:0.132.1
 args: ["--config=/conf/config.yaml"]
 ports:
 - containerPort: 4317
 name: otlp-grpc
 - containerPort: 4318
 name: otlp-http
 volumeMounts:
 - name: config
 mountPath: /conf
 resources:
 requests:
 cpu: 50m
 memory: 128Mi
 limits:
 cpu: 200m
 memory: 256Mi
 volumes:
 - name: config
 configMap:
 name: langfuse-otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
 name: langfuse-otel-collector
 namespace: agentgateway-system
spec:
 selector:
 app: langfuse-otel-collector
 ports:
 - name: otlp-grpc
 port: 4317
 targetPort: 4317
 - name: otlp-http
 port: 4318
 targetPort: 4318

Deploy it:

kubectl apply -f langfuse-collector.yaml

Step 6: Enable Tracing on AgentGateway

For AgentGateway OSS, configure tracing via Helm values. Create values-tracing.yaml:

gateway:
 envs:
 OTEL_EXPORTER_OTLP_ENDPOINT: "http://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317"
 OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"

Upgrade the installation:

helm upgrade agentgateway oci://cr.agentgateway.dev/helm/agentgateway \
 --version 2.1.0 \
 --namespace agentgateway-system \
 -f values-tracing.yaml

Using AgentGateway Enterprise? Instead of environment variables, create an EnterpriseAgentgatewayParameters resource with full control over field mappings:

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayParameters
metadata:
 name: tracing
 namespace: agentgateway-system
spec:
 rawConfig:
 config:
 tracing:
 otlpEndpoint: grpc://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317
 otlpProtocol: grpc
 randomSampling: true
 fields:
 add:
 gen_ai.operation.name: '"chat"'
 gen_ai.system: "llm.provider"
 gen_ai.request.model: "llm.requestModel"
 gen_ai.response.model: "llm.responseModel"
 gen_ai.usage.prompt_tokens: "llm.inputTokens"
 gen_ai.usage.completion_tokens: "llm.outputTokens"
 gen_ai.usage.total_tokens: "llm.totalTokens"
 gen_ai.request.temperature: "llm.params.temperature"
 gen_ai.prompt: "llm.prompt"
 gen_ai.completion: "llm.completion"

Step 7: Create a Gateway and LLM Route

Create the Gateway resource and an OpenAI route:

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: ai-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: llm
 port: 8080
 protocol: HTTPS
 allowedRoutes:
 namespaces:
 from: Same

# openai-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: ai-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - group: agentgateway.dev
 kind: AgentgatewayBackend
 name: openai
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 type: llm
 llm:
 provider:
 openai:
 authToken:
 secretRef:
 name: openai-api-key
 namespace: agentgateway-system

Create the secret and apply everything:

kubectl create secret generic openai-api-key \
 -n agentgateway-system \
 --from-literal=Authorization="Bearer $OPENAI_API_KEY"

kubectl apply -f gateway.yaml
kubectl apply -f openai-route.yaml

Step 8: Test It

Port-forward the gateway and send a request:

kubectl port-forward -n agentgateway-system svc/ai-gateway 8080:8080 &

curl -X POST http://localhost:8080/openai/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4.1-mini",
 "messages": [{"role": "user", "content": "Hello from AgentGateway!"}]
 }'

Step 9: View Traces in Langfuse

Open your Langfuse UI and navigate to Traces. You should see a new trace with the full picture:

Model and provider — which model actually served the request
Token counts — input, output, and total tokens consumed
Full prompt and completion — the exact content sent and received
Gateway metadata — route name, backend endpoint, listener

What Gets Captured

AgentGateway follows the OpenTelemetry GenAI semantic conventions, so every trace includes structured attributes:

Attribute	What It Tells You
`gen_ai.system`	LLM provider (openai, anthropic, etc.)
`gen_ai.request.model`	The model you asked for
`gen_ai.response.model`	The model that actually responded
`gen_ai.usage.prompt_tokens`	Input tokens consumed
`gen_ai.usage.completion_tokens`	Output tokens generated
`gen_ai.prompt`	Full prompt content
`gen_ai.completion`	Full completion content
`gateway`	AgentGateway resource name
`route`	HTTPRoute that matched
`endpoint`	Backend LLM endpoint

Going Further

Fan-Out to Multiple Backends

The OTel Collector makes it easy to send traces to Langfuse and another backend simultaneously:

exporters:
 otlphttp/langfuse:
 endpoint: https://cloud.langfuse.com/api/public/otel
 headers:
 Authorization: "Basic <CREDENTIALS>"
 otlp/jaeger:
 endpoint: jaeger-collector:4317
 tls:
 insecure: true

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlphttp/langfuse, otlp/jaeger]

MCP Tool Tracing

AgentGateway also traces MCP (Model Context Protocol) traffic. When an agent discovers or invokes tools through an MCP server proxied by AgentGateway, you’ll see tool discovery requests, execution calls with parameters and results, and backend server latency — all as spans within the same trace.

Security Policy Visibility

When AgentGateway’s security policies fire — PII protection, prompt injection detection, credential leak prevention — the trace metadata shows which policy triggered, what action was taken (block, mask, allow), and what pattern matched. You get observability into both your LLM interactions and your security guardrails in one place.

Troubleshooting

No traces appearing?

Check the collector is running: kubectl get pods -n agentgateway-system -l app=langfuse-otel-collector
Check collector logs for errors: kubectl logs -n agentgateway-system -l app=langfuse-otel-collector
Wrong API keys show up as 401 errors in collector logs
Make sure proxies were restarted after configuring tracing

Traces visible in gateway logs but not Langfuse?

Langfuse only supports OTLP HTTP, not gRPC — that’s why the collector is needed for protocol conversion
Verify the exporter endpoint URL includes /api/public/otel
Double-check the Base64 credentials format: base64(public_key:secret_key)

Incomplete trace data?

For Enterprise, ensure fields.add includes the GenAI attribute mappings
Set randomSampling: true to capture all requests during testing

Cleanup

kind delete cluster --name agentgateway

Resources

Explore Gateway API and Agentgateway with ExtAuth Match 💕 Game

Fri, 13 Feb 2026 00:00:00 +0000

Over the past few months, we’ve been blown away by the number of contributions and the growing interest in agentgateway. Not only did agentgateway make it onto the CNCF Agentic AI Tech Radar in the Trial category, but the Istio community has also started integrating more deeply with agentgateway after validating its performance and scalability benefits.

With Valentine’s Day around the corner, we started thinking about the amazing contributors and users in our community. What better way to celebrate than by sending some love ❤️ — while having a little fun learning together?

So we built something playful: a swipe-based authorization match game powered by external auth. Think “swipe the card right to allow traffic, swipe left to deny” — but running on Kubernetes with Gateway API and agentgateway.

What is agentgateway?

Agentgateway is a high-performance, lightweight gateway designed specifically for MCP and AI agents — while still working seamlessly with traditional microservices.

There are many gateways available today, but most were designed before the rise of AI agents. As a result, they often struggle to support modern AI protocols without significant rearchitecture. To keep pace with rapid innovation in the age of AI, we needed a purpose-built solution designed specifically for AI-driven workloads. That’s how agentgateway was born.

Since launching agentgateway last year, one of the biggest challenges in Kubernetes has been ensuring that the control plane evolves as quickly as the rapidly changing data plane. In the newly released agentgateway 2.2 version, we re-engineered the control plane and its APIs to enable much faster iteration. This allows us to maintain strong feature parity between standalone and Kubernetes deployments while significantly improving control plane performance.

What You’ll Learn

In this fun Valentine’s Day game, you’ll learn:

How to use the Kubernetes Gateway API (Gateway and HTTPRoute) to manage traffic.
How to extend behavior with agentgateway-specific resources such as AgentGatewayPolicy.
How external authorization works in practice.
How policy decisions dynamically control request flow.

You’ll see it in action, with a human in the loop making real-time decisions.

How the ExtAuth Match Game Works

Here’s the flow:

Visit or refresh the match app and scan the QR code to connect your phone.

Once connected, refresh the match app to trigger a request that enters agentgateway. Alternatively, you can click the Not seeing requests? Try again button. The request is routed using Gateway and HTTPRoute.
Before the request is matched, agentgateway triggers an external authorization check.

The external auth match check appears in a playful swipe card on your phone.
You swipe:
- 👉 Right = allow
- 👈 Left = deny

6. The decision is returned to the gateway, which either approves or denies the match.

You are presented with five cards in total, representing external authorization checks for five different resources.

Under the hood, this demonstrates a production-grade pattern with agentgateway:

External auth policy enforcement
Dynamic request evaluation
Real-time traffic control

It’s a fun, interactive way to understand how agentgateway enforces external auth policy — made with ❤️ by the agentgateway community.

Have Fun! ❤️

If you have a Kubernetes cluster, this Valentine’s Day, we invite you to:

Follow the quick start guide to deploy the game.
Explore Gateway API resources.
Experiment with AgentGatewayPolicy.
Visit/refresh the match app from your browser following the instruction to trigger a request to agentgateway. Swipe the cards left or right on your phone to approve or deny.
Learn something new.
Share your love for Kubernetes Gateway API and agentgateway in the open. Tag @agentgateway when you share on LinkedIn — we’ll select one lucky winner on February 20 for the most fun post.

Got questions? Join us on our discord. We’d love to help with your NGINX ingress migration or support you on your agentic AI journey.

Your First AI Route: Connecting to OpenAI with AgentGateway

Wed, 11 Feb 2026 00:00:00 +0000

Introduction

This is a how-to guide to setup AgentGateway and get your first AI route working with OpenAI. We’ll walk through the complete setup from scratch - creating a Kubernetes cluster, installing AgentGateway, and connecting it to OpenAI’s API.

What is AgentGateway?

AgentGateway is an open source, AI-native data plane built in Rust for connecting, securing, and observing AI traffic. Originally created by Solo.io and now a Linux Foundation project, it acts as a purpose-built proxy layer between your applications and AI services like LLMs, MCP tool servers, and other AI agents.

Traditional API gateways don’t fit AI workloads — AI inference requests are long-running (minutes vs milliseconds), have larger payloads, and can consume entire GPUs, unlike standard web traffic.
Connectivity — Unified interface to route requests to LLM providers (OpenAI, Anthropic, Bedrock, etc.), self-hosted models, and MCP tool servers.
Security — Built-in auth, RBAC, and secrets management for API keys and sensitive data.
Observability — Automatic token counting, cost tracking, and OpenTelemetry-compatible structured logs.
MCP support — Can federate multiple MCP servers behind a single endpoint, and expose legacy REST APIs as MCP tools via OpenAPI integration.
A2A support — Native Agent-to-Agent protocol for secure inter-agent communication..

In this tutorial, we’ll focus on one of AgentGateway’s most common use cases: routing requests to an LLM provider (OpenAI) with secure credential management and built-in cost observability.

What You’ll Learn

Create a Kubernetes cluster and install AgentGateway
Set up secure OpenAI API key storage
Configure AgentGateway to route to OpenAI
Test chat completions, embeddings, and model listings
Monitor real AI requests and track costs
Troubleshoot common issues

Prerequisites

Docker installed and running
kubectl CLI tool
Helm 3.x installed
Valid OpenAI API Key with credits (get from OpenAI Platform)
Basic understanding of Kubernetes and OpenAI API structure

Step 1: Environment Setup

In this step, we’ll create a local Kubernetes cluster using kind (Kubernetes in Docker) and install AgentGateway. This gives us a complete testing environment that mirrors production setups but runs entirely on your local machine.

Install Kind

Kind creates Kubernetes clusters using Docker containers as nodes. This is perfect for development and testing because it’s lightweight, fast to spin up, and doesn’t require cloud resources.

# On macOS 
brew install kind

# On Linux
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind

Create Kind Cluster

This creates a single-node Kubernetes cluster that will host our AgentGateway installation. The cluster provides the foundation for all the networking, security, and routing capabilities we’ll configure.

# Create the cluster
kind create cluster --name agentgateway

# Verify cluster is ready
kubectl get nodes

Install AgentGateway

AgentGateway installation happens in three phases: First we install the Kubernetes Gateway API (the standard for ingress traffic), then AgentGateway’s custom resources, and finally the control plane that manages everything. This separation allows for better modularity and easier upgrades.

# 1. Install Gateway API CRDs (version 1.4.0)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

# 2. Install AgentGateway CRDs
helm upgrade -i --create-namespace \
 --namespace agentgateway-system \
 --version v1.3.11.4.0-alpha.1 agentgateway-crds \
 oci://ghcr.io/kgateway-dev/charts/agentgateway-crds

# 3. Install AgentGateway control plane
helm upgrade -i -n agentgateway-system agentgateway \
 oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --version v1.3.11.4.0-alpha.1

# 4. Verify installation
kubectl get pods -n agentgateway-system

Step 2: OpenAI API Key Setup

Security is paramount when working with AI services. Instead of embedding API keys directly in configurations, we’ll use Kubernetes secrets to store credentials securely. This approach ensures keys are encrypted at rest and can be rotated without changing application code.

Get Your OpenAI API Key

OpenAI uses API keys for authentication and billing. Each key is tied to your account and usage limits, making it essential to secure them properly.

Visit OpenAI Platform
Navigate to API Keys section and create a new key
Set usage limits to control costs
Copy your API key securely

Test Your API Key

Before integrating with AgentGateway, we’ll verify the API key works directly with OpenAI’s API. This eliminates the key as a potential issue if something goes wrong later in the setup.

# Set your OpenAI API key (replace with your actual key)
export OPENAI_API_KEY="sk-your-openai-api-key-here"

# Test the key directly
curl -s "https://api.openai.com/v1/models" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 | jq '.data[0:3] | .[].id'

Create Kubernetes Secret

Kubernetes secrets provide a secure way to store sensitive data like API keys. We format the key as a complete Authorization header (Bearer sk-...) so AgentGateway can use it directly without modification. The --dry-run=client -o yaml | kubectl apply -f - pattern ensures the secret is created safely even if it already exists.

# Create secret with proper authorization header format
kubectl create secret generic openai-secret \
 -n agentgateway-system \
 --from-literal="Authorization=Bearer $OPENAI_API_KEY" \
 --dry-run=client -o yaml | kubectl apply -f -

# Verify secret creation
kubectl get secret openai-secret -n agentgateway-system

Step 3: Configure AgentGateway

Now we’ll configure the core components that make AI routing work. AgentGateway follows the Kubernetes Gateway API pattern with three main resources: Gateway (the entry point), Backends (destination services), and HTTPRoutes (traffic routing rules). This declarative approach makes configurations version-controllable and environment-portable.

Create Gateway Resource

The Gateway resource defines the entry point for all incoming traffic. It specifies which ports to listen on, what protocols to accept, and which namespaces can create routes through it. Think of it as the front door to your AI services.

kubectl apply -f- <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - protocol: HTTP
 port: 8080
 name: http
 allowedRoutes:
 namespaces:
 from: All
EOF

Create OpenAI Backend

AgentgatewayBackend resources define how to connect to AI services. The ai.provider.openai section tells AgentGateway this is an AI service that expects OpenAI-compatible requests. The authentication policy references our secret, and the timeout ensures long-running AI requests don’t hang indefinitely.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai-backend
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: gpt-4o-mini
 policies:
 auth:
 secretRef:
 name: openai-secret
EOF

Create HTTP Routes

Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend. The following example sets up a route. Note that agentgateway automatically rewrites the endpoint to the OpenAI /v1/chat/completions endpoint.

Create the HTTP routes:

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai-chat
 namespace: agentgateway-system
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - name: openai-backend
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Verify Configuration

Before testing, we’ll check that all our resources are properly created and accepted by the AgentGateway controller. The Accepted status indicates that configurations are valid and the controller can proceed with implementation.

# Check both backends
kubectl get agentgatewaybackend -n agentgateway-system

# Check routes
kubectl get httproute -n agentgateway-system

# Check Gateway
kubectl get gateway agentgateway-proxy -n agentgateway-system

Step 4: Testing Your Setup

With our configuration complete, it’s time to test the AI routing. Since we’re using a local kind cluster, we’ll use port-forwarding to access the Gateway service. In production, this would be handled by a LoadBalancer or Ingress controller.

Setup Port-Forward

Port-forwarding creates a tunnel from your local machine to the AgentGateway service inside the Kubernetes cluster. This lets us test the setup without exposing services publicly.

# Port-forward AgentGateway service in background
kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:8080 &

Test Chat Completions

This is where the magic happens! Our request travels through AgentGateway, gets authenticated using our secret, routed to OpenAI’s API, and returns with a complete AI response. Notice how the response includes token usage information that AgentGateway automatically captures for cost tracking and observability.

# Test basic chat completion
curl -i "localhost:8080/openai/chat/completions" \
 -H "content-type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [
 {
 "role": "user",
 "content": "What are the key benefits of using an AI Gateway?"
 }
 ],
 "max_tokens": 100
 }'

Expected Response:

{
 "id": "chatcmpl-abc123def456",
 "object": "chat.completion",
 "created": 1701234567,
 "model": "gpt-4o-mini-2024-07-18",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "An AI Gateway provides unified access to multiple AI providers, centralized security and authentication, comprehensive observability and cost tracking, rate limiting and quotas, and improved reliability through failover and retry mechanisms."
 },
 "finish_reason": "stop"
 }
 ],
 "usage": {
 "prompt_tokens": 15,
 "completion_tokens": 35,
 "total_tokens": 50
 }
}

Test Different Models

AgentGateway allows you to easily switch between different OpenAI models by simply changing the model parameter. The backend automatically routes to the appropriate model while maintaining consistent authentication and observability.

# Test with GPT-4o
curl -s "localhost:8080/openai/chat/completions" \
 -H "content-type: application/json" \
 -d '{
 "model": "gpt-4o",
 "messages": [
 {
 "role": "user",
 "content": "Explain AgentGateway in one sentence."
 }
 ],
 "max_tokens": 50
 }' | jq '.choices[0].message.content'

Troubleshooting

When working with distributed systems like Kubernetes and external APIs, issues can arise at multiple layers. This section covers the most common problems you might encounter and how to systematically diagnose them. The key is to test each layer independently: network connectivity, authentication, resource configuration, and API compatibility.

Common Issues

1. Service Not Found Error: This usually means the service name doesn’t match what was actually created during installation. Different AgentGateway versions or installation methods may create services with different names.

# Check what services exist
kubectl get svc -n agentgateway-system

# If agentgateway-proxy doesn't exist, use the correct service name
kubectl get svc -n agentgateway-system | grep -i gateway

2. Authentication Errors (401): Authentication failures typically indicate either an invalid API key or incorrect secret formatting. Always test the key directly with OpenAI before troubleshooting AgentGateway.

# Verify secret exists
kubectl get secret openai-secret -n agentgateway-system -o yaml

# Test API key directly
curl -s "https://api.openai.com/v1/models" \
 -H "Authorization: Bearer $OPENAI_API_KEY" | jq '.data[0].id'

3. Routes Not Working: Route issues often stem from mismatched resource names or namespaces. The Gateway, HTTPRoute, and Backend must all reference each other correctly for traffic to flow.

# Check backend status
kubectl describe agentgatewaybackend openai-backend -n agentgateway-system

# Check route status
kubectl describe httproute openai-chat -n agentgateway-system

# Check Gateway status
kubectl describe gateway agentgateway-proxy -n agentgateway-system

4. Port-Forward Issues: Port conflicts are common on development machines. If port 8080 is busy, either stop the conflicting service or use a different port.

# Check if port 8080 is in use
lsof -i :8080

# Try a different port
kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8081:8080 &
export GATEWAY_PORT="8081"

Debug Commands

# View all AgentGateway resources
kubectl get agentgatewaybackend,gateway,httproute -n agentgateway-system

# Check pod logs for errors
kubectl logs deploy/agentgateway -n agentgateway-system --tail=20

# Test connectivity from inside cluster
kubectl exec -n agentgateway-system deploy/agentgateway -- \
 curl -v https://api.openai.com/v1/models \
 -H "Authorization: Bearer $OPENAI_API_KEY"

Cleanup

When you’re done experimenting, it’s important to clean up resources to free up system resources and avoid any potential costs. The cleanup process should happen in reverse order: stop network connections first, then remove application resources, and finally remove infrastructure.

Stop Port-Forward

# Kill the port-forward process
kill $PORTFORWARD_PID

Remove Resources (Optional)

This removes all the AgentGateway configuration we created, but leaves the AgentGateway installation intact for future experiments. Remove resources in dependency order: routes first (they reference backends), then backends, then the Gateway.

# Remove all OpenAI configuration
kubectl delete httproute openai-chat openai-models -n agentgateway-system
kubectl delete agentgatewaybackend openai-backend openai-models-backend -n agentgateway-system
kubectl delete gateway agentgateway-proxy -n agentgateway-system
kubectl delete secret openai-secret -n agentgateway-system

Remove Kind Cluster (Optional)

This completely removes the Kubernetes cluster and all associated resources. Only do this if you’re completely done with the tutorial, as you’ll need to recreate everything from Step 1 to run it again.

# Delete the entire cluster
kind delete cluster --name agentgateway

Next Steps

Now that you have a working AI gateway, you can build on this foundation to create production-ready AI infrastructure:

Add more providers - Configure Anthropic, AWS Bedrock, or Azure OpenAI for multi-provider setups and failover scenarios
Implement security - Add rate limiting, authentication, and guardrails to protect against abuse and unexpected costs
Set up monitoring - Configure Grafana dashboards and alerting to track performance, costs, and usage patterns across teams
Explore advanced routing - Implement path-based, header-based, and weighted routing to direct different types of requests to optimal models

Key Takeaways

This tutorial demonstrates several important concepts for production AI systems:

AgentGateway provides a unified interface to AI providers with minimal overhead, making it easy to switch providers or implement failover
Proper secret management is essential for production deployments - never embed API keys in code or configuration files
Built-in observability gives immediate insights into costs and performance without requiring additional tooling or instrumentation
The Gateway API pattern makes routing configuration declarative and portable across different Kubernetes environments
Dual backend types (AI-aware vs static HTTP) allow you to handle both complex AI workloads and simple metadata requests efficiently
Kind clusters are perfect for local development and testing, providing a production-like environment without cloud costs

Your AgentGateway is now successfully routing requests to OpenAI with enterprise-grade security, observability, and cost control! You’ve built a foundation that can scale from development to production while maintaining visibility and control over your AI infrastructure. 🎯

Getting started with Multi-LLM provider routing

Mon, 09 Feb 2026 00:00:00 +0000

How to: Building Agentgateway to support Multi-LLM providers.

Agentgateway makes it simple to route traffic to multiple LLM providers through a single gateway using the Kubernetes Gateway API. This guide walks through setting up agentgateway OSS on a local Kind cluster with xAI, Anthropic, and OpenAI backends, all routed through a listener named llm-providers.

One of the most common patterns in AI-native infrastructure is routing traffic to multiple LLM providers behind a single entry point. Whether you’re comparing models, building failover strategies, or just want a unified API across providers, agentgateway gives you a clean Kubernetes-native way to do it using the Gateway API and AgentgatewayBackend custom resources.

In this guide, we’ll set up a complete working example on a local Kind cluster with three LLM providers routed via path-based HTTPRoute resources.

What you’ll build

By the end of this guide you’ll have:

A Kind cluster running the agentgateway control plane
A Gateway with a listener named llm-providers on port 8080
Three AgentgatewayBackend resources for xAI, Anthropic, and OpenAI
Three HTTPRoute resources that route /xai, /anthropic, and /openai to their respective backends

Prerequisites

Before getting started, make sure you have the following installed:

Docker — container runtime for Kind
Kind — local Kubernetes clusters
kubectl — Kubernetes CLI (within 1 minor version of your cluster)
Helm — Kubernetes package manager

You also need API keys for the LLM providers you want to use. Export them as environment variables:

export XAI_API_KEY="your-xai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export OPENAI_API_KEY="your-openai-api-key"

Step 1: Create a Kind cluster

Create a local Kubernetes cluster using Kind. This gives you a lightweight, disposable cluster perfect for testing.

kind create cluster --name agentgateway-demo

Verify it’s running:

kubectl cluster-info --context kind-agentgateway-demo
kubectl get nodes

Step 2: Install agentgateway OSS via Helm

Install the Gateway API CRDs

Agentgateway relies on the Kubernetes Gateway API. Install the standard CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

Install the agentgateway CRDs

Install the custom resource definitions that agentgateway needs (AgentgatewayBackend, AgentgatewayPolicy, etc.):

helm upgrade -i agentgateway-crds \
 oci://ghcr.io/kgateway-dev/charts/agentgateway-crds \
 --create-namespace \
 --namespace agentgateway-system \
 --version v1.3.11.4.0-alpha.1

Install the agentgateway control plane

helm upgrade -i agentgateway \
 oci://ghcr.io/kgateway-dev/charts/agentgateway \
 --namespace agentgateway-system \
 --version v1.3.11.4.0-alpha.1 \
 --set controller.image.pullPolicy=Always

The --set controller.image.pullPolicy=Always flag is recommended for development builds to ensure you always get the latest image.

Verify the pods are running:

kubectl get pods -n agentgateway-system

You should see the controller pod in a Running state.

Step 3: Create the Gateway

The Gateway resource is the entry point for all traffic. It defines a listener named llm-providers on port 8080 that accepts HTTPRoute resources from any namespace.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: agentgateway-proxy
 namespace: agentgateway-system
spec:
 gatewayClassName: enterprise-agentgateway
 infrastructure:
 parametersRef:
 name: tracing
 group: enterpriseagentgateway.solo.io
 kind: EnterpriseAgentgatewayParameters
 listeners:
 - protocol: HTTP
 port: 8080
 name: llm-providers
 allowedRoutes:
 namespaces:
 from: All
EOF

The listener name llm-providers is the key here. All HTTPRoute resources in the following steps reference this listener via sectionName, so the gateway knows which listener should handle each route.

Step 4: Configure API key secrets

Each LLM provider needs an API key stored as a Kubernetes Secret. The AgentgatewayBackend resources reference these secrets for authentication.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: xai-secret
 namespace: agentgateway-system
type: Opaque
stringData:
 Authorization: $XAI_API_KEY
EOF

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: anthropic-secret
 namespace: agentgateway-system
type: Opaque
stringData:
 Authorization: $ANTHROPIC_API_KEY
EOF

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: openai-secret
 namespace: agentgateway-system
type: Opaque
stringData:
 Authorization: $OPENAI_API_KEY
EOF

Never commit API keys to source control. Use environment variable substitution or a secrets manager in production.

Step 5: Create agentgateway backends

AgentgatewayBackend resources define the LLM provider endpoints. Each backend specifies the provider type, model, and authentication. Agentgateway automatically rewrites requests to the correct chat completion endpoint for each provider.

xAI backend

xAI uses an OpenAI-compatible API. Because we’re specifying a custom host (api.x.ai) rather than the default OpenAI host, we need to explicitly set the host, port, path, and TLS SNI.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: xai
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: grok-4-1-fast-reasoning
 host: api.x.ai
 port: 443
 path: "/v1/chat/completions"
 policies:
 auth:
 secretRef:
 name: xai-secret
 tls:
 sni: api.x.ai
EOF

Anthropic backend

Anthropic uses its native provider type. Agentgateway handles the endpoint rewriting automatically — no custom host or TLS configuration needed.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: anthropic
 namespace: agentgateway-system
spec:
 ai:
 provider:
 anthropic:
 model: "claude-sonnet-4-5-20250929"
 policies:
 auth:
 secretRef:
 name: anthropic-secret
EOF

OpenAI backend

OpenAI also uses its native provider type with the default endpoint.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
 name: openai
 namespace: agentgateway-system
spec:
 ai:
 provider:
 openai:
 model: gpt-4o-mini
 policies:
 auth:
 secretRef:
 name: openai-secret
EOF

Step 6: Create HTTPRoutes

HTTPRoute resources connect incoming request paths to the AgentgatewayBackend resources. Each route references the llm-providers listener on the Gateway via sectionName, and matches a path prefix to direct traffic to the correct backend.

Route	Path	Backend	Provider
xai	`/xai`	xai	xAI (Grok)
anthropic	`/anthropic`	anthropic	Anthropic (Claude)
openai	`/openai`	openai	OpenAI (GPT)

xAI route

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: xai
 namespace: agentgateway-system
 labels:
 route-type: llm-provider
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 sectionName: llm-providers
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /xai
 backendRefs:
 - name: xai
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

Anthropic route

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: anthropic
 namespace: agentgateway-system
 labels:
 route-type: llm-provider
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 sectionName: llm-providers
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /anthropic
 backendRefs:
 - name: anthropic
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

OpenAI route

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: openai
 namespace: agentgateway-system
 labels:
 route-type: llm-provider
spec:
 parentRefs:
 - name: agentgateway-proxy
 namespace: agentgateway-system
 sectionName: llm-providers
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /openai
 backendRefs:
 - name: openai
 namespace: agentgateway-system
 group: agentgateway.dev
 kind: AgentgatewayBackend
EOF

The key fields that tie everything together:

parentRefs.sectionName: llm-providers — binds the route to the specific Gateway listener
backendRefs.group: agentgateway.dev — tells the Gateway API to look for AgentgatewayBackend resources (not standard Kubernetes Service objects)
backendRefs.kind: AgentgatewayBackend — references the custom backend type
labels.route-type: llm-provider — optional label useful for filtering and grouping

Step 7: Verify and test

Once all resources are applied, verify everything is connected and working.

Check resource status

# Verify the Gateway is accepted
kubectl get gateway -n agentgateway-system

# Verify backends exist
kubectl get agentgatewaybackend -n agentgateway-system

# Verify routes are attached
kubectl get httproute -n agentgateway-system

Port-forward and test

Forward the gateway port to your local machine and send a test request:

kubectl port-forward -n agentgateway-system \
 svc/agentgateway-proxy 8080:8080 &

Test the OpenAI route:

curl -s http://localhost:8080/openai \
 -H "Content-Type: application/json" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [{"role": "user", "content": "Hello"}]
 }'

Test the Anthropic route:

curl -s http://localhost:8080/anthropic \
 -H "Content-Type: application/json" \
 -d '{
 "model": "claude-sonnet-4-5-20250929",
 "messages": [{"role": "user", "content": "Hello"}]
 }'

Test the xAI route:

curl -s http://localhost:8080/xai \
 -H "Content-Type: application/json" \
 -d '{
 "model": "grok-4-1-fast-reasoning",
 "messages": [{"role": "user", "content": "Hello"}]
 }'

Agentgateway automatically rewrites requests to each provider’s chat completion endpoint, so you use a unified request format regardless of the backend provider.

Cleanup

When you’re done, remove everything:

# Remove routes and backends
kubectl delete httproute xai anthropic openai -n agentgateway-system
kubectl delete agentgatewaybackend xai anthropic openai -n agentgateway-system
kubectl delete secret xai-secret anthropic-secret openai-secret -n agentgateway-system
kubectl delete gateway agentgateway-proxy -n agentgateway-system

# Uninstall Helm charts
helm uninstall agentgateway agentgateway-crds -n agentgateway-system

# Delete the Kind cluster
kind delete cluster --name agentgateway-demo

What’s next

Now that you have path-based LLM routing working, there’s a lot more you can do with agentgateway:

Multiple providers on one route — group backends for automatic load balancing and failover. Agentgateway picks two random providers and selects the healthiest one.
Prompt guarding — add AgentgatewayPolicy resources for regex-based prompt filtering or webhook-based validation before requests hit your LLM.
Rate limiting — protect your API keys and budgets with local or remote rate limiting policies.
Observability — enable full OpenTelemetry support for metrics, logs, and distributed tracing across all your LLM traffic.

Check out the agentgateway docs for more, or come chat with us on Discord.

With a single Gateway listener and a few YAML resources, you get a unified, Kubernetes-native control point for all your LLM traffic. That’s the power of agentgateway.

agentgateway adds support for migrating from ingress-nginx

Fri, 30 Jan 2026 00:00:00 +0000

ingress-nginx retirement

In November 2025 the Kubernetes community announced the retirement of the venerable ingress-nginx Ingress controller.

Ingress-nginx was one of the first options available in Kubernetes for configuring ingress traffic to workloads hosted on Kubernetes. The story of ingress and traffic management has evolved steadily over the years, culminating in the Kubernetes Gateway API specification, which offers a more robust and complete solution to the problem.

Support for ingress-nginx is scheduled to end in March.

Help is on the way!

The kgateway and agentgateway project maintainers extended the ingress2gateway migration tool to support easily migrating from ingress-nginx to their respective projects.

This article dives into the use of this tool to migrate to agentgateway. You will find the tool’s installation instructions in the project’s documentation.

How to use the tool

The tool’s main subcommand is the print command which accepts both a providers flag and an emitter flag: the “from” and “to” formats for performing the translation to Gateway API.

ingress2gateway print --help

Here is the help output for the print command:

Prints Gateway API objects generated from ingress and provider-specific resources.
Usage:
ingress2gateway print [flags]
Flags:
-A, --all-namespaces If present, list the requested objects across all namespaces. Namespace in current context is ignored even
if specified with --namespace.
--emitter standard If present, the tool will try to use the specified emitter to generate the Gateway API resources, supported values are [agentgateway kgateway standard]. The standard emitter will only output Gateway API (default "standard")
-h, --help help for print
--ingress-nginx-ingress-class string Provider-specific: ingress-nginx. The name of the ingress class to select. Defaults to 'nginx' (default "nginx")
--input-file string Path to the manifest file. When set, the tool will read ingresses from the file instead of reading from the cluster. Supported files are yaml and json.
-n, --namespace string If present, the namespace scope for this CLI request.
-o, --output string Output format. One of: (yaml, json, kyaml). (default "yaml")
--providers strings If present, the tool will try to convert only resources related to the specified providers, supported values are [ingress-nginx].
Global Flags:
--kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.

Besides providers and emitter, there are flags to control the source of the input: whether from a file (input-file), or by looking at the Kubernetes cluster, either in --all-namespaces or a specific --namespace.

ingress2gateway supports both kgateway and agentgateway

Through the emitter flag, you can target either kgateway (emitter=kgateway) or agentgateway (emitter=agentgateway).

What’s special about the tool is its support for a litany of ingress-nginx annotations for configuring a variety of features including CORS, rate limiting, canaries, timeouts, auth, session affinity, etc..

This version of ingress2gateway gives you the ability to automatically translate your ingress-nginx configurations to agentgateway, and it knows to configure agentgateway-specific resources as necessary to translate the configuration including the ingress-nginx annotations.

In a previous blog Michael Levan provided an overview of how to work with the kgateway emitter to translate configurations for three distinct scenarios: TLS, Auth, and CORS.

In this blog, we walk you through the same three scenarios, but this time targeting the agentgateway emitter.

Let’s get started!

Setup

Provision a test Kubernetes cluster. A local cluster such as kind or k3d will do, or feel free to use one provisioned by your favorite cloud provider.

Install kgateway with agentgateway

Install agentgateway per the install instructions.

Once installed, confirm that the agentgateway control plane is running in the namespace agentgateway-system:

kubectl get pod -n agentgateway-system

NAME READY STATUS RESTARTS AGE
agentgateway-7455b4475b-x9sxt 1/1 Running 0 11h

Deploy a sample backend workload

Deploy the httpbin service (configured to use port 8000):

kubectl apply -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/httpbin/httpbin.yaml

Verify that the httpbin pod is running and its service exists in the default namespace:

kubectl get pod,svc

Scenario 1: TLS

Review the following initial Ingress configuration, which configures ingress with TLS termination for httpbin:

cat tls.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: tls-ingress
 annotations:
 nginx.ingress.kubernetes.io/ssl-redirect: "true"
 nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
 ingressClassName: nginx
 tls:
 - hosts:
 - httpbin.example.com
 secretName: httpbin-cert
 rules:
 - host: httpbin.example.com
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: httpbin
 port:
 number: 8000

Run ingress2gateway to generate the associated Gateway API configuration translation:

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=tls.yaml

Here is the generated output:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: nginx
spec:
 gatewayClassName: agentgateway
 listeners:
 - hostname: httpbin.example.com
 name: httpbin-example-com-http
 port: 80
 protocol: HTTP
 - hostname: httpbin.example.com
 name: httpbin-example-com-https
 port: 443
 protocol: HTTPS
 tls:
 certificateRefs:
 - group: null
 kind: null
 name: httpbin-cert
status: {}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: tls-ingress-httpbin-example-com-http-redirect
spec:
 hostnames:
 - httpbin.example.com
 parentRefs:
 - name: nginx
 sectionName: httpbin-example-com-http
 rules:
 - filters:
 - requestRedirect:
 scheme: https
 statusCode: 301
 type: RequestRedirect
 matches:
 - path:
 type: PathPrefix
 value: /
status:
 parents: []
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: tls-ingress-httpbin-example-com-https
spec:
 hostnames:
 - httpbin.example.com
 parentRefs:
 - name: nginx
 sectionName: httpbin-example-com-https
 rules:
 - backendRefs:
 - name: httpbin
 port: 8000
 matches:
 - path:
 type: PathPrefix
 value: /
status:
 parents: []

Note the Gateway and HTTPRoute’s in the output. Two HTTPRoute resources are created, one to route requests and the other to redirect HTTP requests to the HTTPS scheme.

Apply the resources to the cluster

Since we are configuring TLS, we need a certificate:

step certificate create httpbin.example.com httpbin.crt httpbin.key \
 --profile self-signed --subtle --no-password --insecure

Create a Kubernetes secret to hold the certificate:

kubectl create secret tls httpbin-cert --cert=httpbin.crt --key=httpbin.key

Apply the generated Gateway API resources to the cluster:

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=tls.yaml | kubectl apply -f -

Test it

Capture the gateway’s external IP address:

export GW_IP=$(kubectl get gateway nginx -o jsonpath='{.status.addresses[0].value}')

Verify that ingress is working with https:

curl -s --insecure https://httpbin.example.com/get --resolve httpbin.example.com:443:$GW_IP

Verify that a request to port 80 is redirected (301) to port 443:

curl -s --head http://httpbin.example.com/get --resolve httpbin.example.com:80:$GW_IP

Here is the output:

HTTP/1.1 301 Moved Permanentlylocation: https://httpbin.example.com/getdate: Tue, 27 Jan 2026 23:25:33 GMT

Cleanup

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=tls.yaml | kubectl delete -f -

Scenario 2: Basic Authentication

Review the initial Ingress resource, which configures ingress with basic authentication for httpbin:

cat basic-auth.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: auth-ingress
 annotations:
 nginx.ingress.kubernetes.io/auth-type: basic
 nginx.ingress.kubernetes.io/auth-secret: basic-auth
 nginx.ingress.kubernetes.io/auth-secret-type: auth-file
spec:
 ingressClassName: nginx
 rules:
 - host: httpbin.example.com
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: httpbin
 port:
 number: 8000

Review the ingress2gateway-generated Gateway API-conformant translation:

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=basic-auth.yaml

Inspect the generated resources:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: nginx
spec:
 gatewayClassName: agentgateway
 listeners:
 - hostname: httpbin.example.com
 name: httpbin-example-com-http
 port: 80
 protocol: HTTP
status: {}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: auth-ingress-httpbin-example-com
spec:
 hostnames:
 - httpbin.example.com
 parentRefs:
 - name: nginx
 rules:
 - backendRefs:
 - name: httpbin
 port: 8000
 matches:
 - path:
 type: PathPrefix
 value: /
status:
 parents: []
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: auth-ingress
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: auth-ingress-httpbin-example-com
 traffic:
 basicAuthentication:
 secretRef:
 name: basic-auth
status:
 ancestors: null

Above we see the generation of three resources: the Gateway, the HTTPRoute, and an AgentGatewayPolicy which captures the basic authentication configuration.

Apply the resources to the cluster

For basic authentication, we need to first create a .htaccess file with the htpasswd command (when prompted for a password enter admin):

htpasswd -c .htaccess admin

Create a Kubernetes secret containing the basic authentication credentials:

kubectl create secret generic basic-auth --from-file=.htaccess

Apply the generated Gateway API resources to the cluster:

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=basic-auth.yaml | kubectl apply -f -

We should now have a Gateway, an HTTPRoute and an AgentGatewayPolicy.

Note the notification from the tool points out that the credentials in the secret need to be accessed via the key .htaccess. This is important since ingress-nginx uses a different key (auth).

Test it

Capture the gateway’s external IP address:

export GW_IP=$(kubectl get gateway nginx -o jsonpath='{.status.addresses[0].value}')

Test 1: should return HTTP 401 Unauthorized when no credentials supplied:

curl --head http://httpbin.example.com/ --resolve httpbin.example.com:80:$GW_IP

Here is the output:

HTTP/1.1 401 Unauthorized
content-type: text/plain
www-authenticate: Basic realm="Restricted"
content-length: 71
date: Tue, 27 Jan 2026 23:34:56 GMT

Test 2: should return HTTP 200 when properly authenticating:

curl --head -u "admin:admin" http://httpbin.example.com/ --resolve httpbin.example.com:80:$GW_IP

Here is the output:

HTTP/1.1 200 OK
access-control-allow-credentials: true
access-control-allow-origin: *
content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
content-type: text/html; charset=utf-8
date: Tue, 27 Jan 2026 23:35:50 GMT

Cleanup

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=basic-auth.yaml | kubectl delete -f -

Scenario 3: CORS

Review the initial Ingress resource, which configures CORS for httpbin:

cat cors.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: cors-ingress
 annotations:
 nginx.ingress.kubernetes.io/enable-cors: "true"
 nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com,https://dashboard.example.com"
 nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST,PUT,DELETE,OPTIONS"
 nginx.ingress.kubernetes.io/cors-allow-headers: "Authorization,Content-Type,X-Requested-With"
 nginx.ingress.kubernetes.io/cors-expose-headers: "X-Custom-Header,X-Request-ID"
 nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
 nginx.ingress.kubernetes.io/cors-max-age: "7200"
spec:
 ingressClassName: nginx
 rules:
 - host: httpbin.example.com
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: httpbin
 port:
 number: 8000

Review the ingress2gateway-generated Gateway API-conformant translation:

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=cors.yaml

Review the generated resources:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: nginx
spec:
 gatewayClassName: agentgateway
 listeners:
 - hostname: httpbin.example.com
 name: httpbin-example-com-http
 port: 80
 protocol: HTTP
status: {}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 annotations:
 gateway.networking.k8s.io/generator: ingress2gateway-v0.2.0-72-g9594e9a
 name: cors-ingress-httpbin-example-com
spec:
 hostnames:
 - httpbin.example.com
 parentRefs:
 - name: nginx
 rules:
 - backendRefs:
 - name: httpbin
 port: 8000
 filters:
 - responseHeaderModifier:
 remove:
 - Access-Control-Allow-Origin
 - Access-Control-Allow-Methods
 - Access-Control-Allow-Headers
 - Access-Control-Expose-Headers
 - Access-Control-Max-Age
 - Access-Control-Allow-Credentials
 type: ResponseHeaderModifier
 matches:
 - path:
 type: PathPrefix
 value: /
status:
 parents: []
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
 name: cors-ingress
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: cors-ingress-httpbin-example-com
 traffic:
 cors:
 allowCredentials: true
 allowHeaders:
 - Authorization
 - Content-Type
 - X-Requested-With
 allowMethods:
 - GET
 - POST
 - PUT
 - DELETE
 - OPTIONS
 allowOrigins:
 - https://app.example.com
 - https://dashboard.example.com
 exposeHeaders:
 - X-Custom-Header
 - X-Request-ID
 maxAge: 7200
status:
 ancestors: null

Above, we see three generated resources. Besides the Gateway, the HTTPRoute and AgentGatewayPolicy configure CORS.

Apply the resources to the cluster

Apply the generated Gateway API resources to the cluster:

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=cors.yaml | kubectl apply -f -

Test it

Capture the gateway’s external IP address:

export GW_IP=$(kubectl get gateway nginx -o jsonpath='{.status.addresses[0].value}')

When testing CORS, we expect both a preflight request (OPTIONS method) and a normal request from a valid origin (e.g. from app.example.com) to result in a response from a server that contains the access-control-allow-origin confirming that the request was from a valid origin:

curl -v http://httpbin.example.com/ --resolve httpbin.example.com:80:$GW_IP \
 -X OPTIONS \
 -H "Origin: https://app.example.com" \
 -H "Access-Control-Request-Method: POST" \
 -H "Access-Control-Request-Headers: Authorization,Content-Type"

Note the access-control-allow-origin response header in the output below:

> OPTIONS / HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/8.18.0
> Accept: */*
> Origin: https://app.example.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: Authorization,Content-Type
>
* Request completely sent off
< HTTP/1.1 200 OK
< access-control-allow-origin: https://app.example.com
< access-control-allow-methods: GET,POST,PUT,DELETE,OPTIONS
< access-control-allow-headers: authorization,content-type,x-requested-with
< access-control-max-age: 7200
< content-length: 0
< date: Tue, 27 Jan 2026 17:22:47 GMT

curl -sv http://httpbin.example.com/get --resolve httpbin.example.com:80:$GW_IP \
 -H "Origin: https://app.example.com" -o /dev/null

The response headers here match the preflight response.

A negative test should return a similar response, but with either no access-control-allow-origin header, or one whose value does not include the origin from which the request was made:

curl -v http://httpbin.example.com/ --resolve httpbin.example.com:80:$GW_IP \
 -X OPTIONS \
 -H "Origin: https://evil.com" \
 -H "Access-Control-Request-Method: POST" \
 -H "Access-Control-Request-Headers: Content-Type,Authorization"

In the below output, note the absence of the access-control-allow-origin header:

> OPTIONS / HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/8.18.0
> Accept: */*
> Origin: https://evil.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: Content-Type,Authorization
>
* Request completely sent off
< HTTP/1.1 200 OK
< date: Tue, 27 Jan 2026 23:40:36 GMT
< content-length: 0

A normal request from a disallowed origin likewise returns a response with the absence of access-control-allow-origin header:

curl -sv http://httpbin.example.com/ --resolve httpbin.example.com:80:$GW_IP \
 -H "Origin: https://evil.com" -o /dev/null

Cleanup

ingress2gateway print --providers=ingress-nginx --emitter=agentgateway --input-file=cors.yaml | kubectl delete -f -

Summary

We invite you to go further and review the project’s migration documentation which covers an even wider range of scenarios ranging from basic ingress, to session affinity, rate limiting, and more.

The agentgateway project has full conformance with the Gateway API. The companion tool ingress2gateway makes short work of translating your existing ingress-nginx Ingress configurations to agentgateway configuration resources, with support for all of these scenarios.

If you have any questions or need help migrating, we’re here to help. You will find all of the documentation, community and other support pages directly from the agentgateway website.

Enterprise MCP SSO With Microsoft Entra and Agentgateway

Mon, 26 Jan 2026 00:00:00 +0000

MCP servers are cropping up all over the enterprise like weeds in a nice lawn. And just like weeds, this can cause problems. MCP servers should be secured, but how? The official spec says use OAuth, but that is not appropriate within an enterprise organization.

On a recent LinkedIn post I made the point:

Any internal enterprise MCP client / AI agent that communicates to an [remote] MCP server should be secured with enterprise SSO. If the agent is acting autonomously, then agent identity should be enforced. But that is for a different post… (check out my 5 part series on Entra Agent ID).

A lot of enterprise organizations use Microsoft Entra ID for their enterprise IdP. In this blog, we take a look at securing ALL MCP access with Entra ID SSO.

SSO has been around for a long time, so what’s the big difference here? They main thing is SSO is usually done in browser based applications. MCP clients are not usually browser applications. For example, VS code, Cursor, Claude, and other AI agents can run on a desktop, mobile, or cloud environment. A browser may be “around” but they are not always the direct interface. So for us to accomplish SSO from an MCP client to an MCP server, the MCP protocol must support it.

OIDC is based on OAuth, and we can perform OIDC logins by leveraging the MCP Authorization spec. And we can do this consistently for all agents/MCP clients regardless of what the backend MCP server is (ie, running within enterprise, hosted by a vendor, SaaS, etc). By tying to enterprise SSO we can apply policy to the MCP usage before it reaches the MCP server. Is this client allowed to access this MCP server? What tools are they allowed to see?

Agentgateway is a powerful opensource (Linux Foundation) MCP gateway that implements the MCP Authorization spec. That means we don’t have to try and re-write all MCP servers to use Entra. We can do it automatically from an MCP gateway.

Let’s look at an example configuration for Agentgateway:

mcpAuthentication:
 mode: strict
 issuer: https://sts.windows.net/${ENTRA_TENANT_ID}/
 jwks:
 url: https://login.microsoftonline.com/${ENTRA_TENANT_ID}/discovery/v2.0/keys
 audiences:
 - api://b92d6e60-86ff-4359-b971-04404fe079ec
 resourceMetadata:
 authorizationServers:
 - https://login.microsoftonline.com/${ENTRA_TENANT_ID}/v2.0 
 resource: https://ceposta-agw.ngrok.io/entra/mcp
 scopesSupported:
 - api://b92d6e60-86ff-4359-b971-04404fe079ec/mcp_access
 # - openid
 # - profile
 bearerMethodsSupported:
 - header
 - body
 - query
 resourceDocumentation: https://ceposta-agw.ngrok.io/entra/mcp/docs
 resourcePolicyUri: https://ceposta-agw.ngrok.io/entra/mcp/policies

This piece of YAML is all that’s needed to implement the MCP Authorization spec to force SSO flows to ANY MCP server hosted on the gateway.

The real detail is in how we configure Entra for this.

Digging into Entra ID for MCP SSO

To use Entra ID for this, we need to set up an Entra App Registration. This will be used to represent our Agentgateway. Can the MCP client access our Agentgateway? We want the Entra ID tokens to be scoped for this. Any other complex authorization policy can be handled with Agentgateway and potentially calling out to a ReBAC engine like OpenFGA.

This Entra stuff takes some attention to detail, so let’s follow it step by step setting up an App Registration for Agentgateway.

Step 1. Create App Registration

Click “Create Application” to begin the registration process. We won’t add any redirect URIs since this app is just used to represent the Agentgateway application, it won’t be doing any oauth flows itself. The MCP clients do that.

Step 2. App Registration Configured

Once the application registration is created, you can see things like it’s service principal / client_id (b92d6e60-86ff-4359-b971-04404fe079ec in our case). You don’t need to add any credentials to this app, but we do want to configure scopes that can be requested so that Entra can correctly configure the aud claim in any tokens it issues.

Step 3. Add Scopes

Click “Expose an API” –> “Add a Scope”:

Step 4. Configure Scope for Agentgateway access

Adding a new scope called mcp_access, leave it as Admin consent and fill in some details that would be displayed on any consent screens. To request the scope, you use the full scope name: api://b92d6e60-86ff-4359-b971-04404fe079ec/mcp_access

Step 5. (Optional - Recommended) add pre-consented Clients

Lastly, we can configure which clients are allowed to request these scopes. For example, we can add the public VS Code client (ie, it’s baked into VS code): aebc6443-996d-45c2-90f0-388ff96faa56.

Testing MCP SSO + Entra

We can run our agentgateway (see source code):

agentgateway -f ./config/agentgateway.yaml

And if we go to VS Code, we can add our new server:

Step 1. Add MCP Server

Step 2. Configure Streamable HTTP MCP Server

Step 3. Configure MCP URL

Step 4. Agree to Perform SSO Login

Step 5. Review MCP Config in `mcp.json`

Using your own MCP Clients

In VS Code (and Cursor, Claude, etc) the MCP client ID is baked in. But if you have your own MCP clients, or AI agents, or just want to configure your own OAuth clients for MCP access, you can do that in the Entra dashboard.

Step 1. Create a new App Registration (Public client is fine)

Step 2. Configure correct Redirect URIs

At this point you can use your own OAuth client_ids.

Caveat for MCP Inspector

MCP Inspector very closely follows the MCP Authorization spec, and unfortunately for Entra ID this causes some issues.

For example, if we take a look at our OAuth Protected Resource Metadata (PRM):

{
 "resource": "https://ceposta-agw.ngrok.io/entra/mcp",
 "authorization_servers": [
 "https://login.microsoftonline.com/5e7d8166-7876-4755-a1a4-b476d4a344f6/v2.0"
 ],
 "scopes_supported": [
 "api://b92d6e60-86ff-4359-b971-04404fe079ec/mcp_access"
 ],
 "bearer_methods_supported": [
 "header",
 "body",
 "query"
 ],
 "resource_documentation": "https://ceposta-agw.ngrok.io/entra/mcp/docs",
 "resource_policy_uri": "https://ceposta-agw.ngrok.io/entra/mcp/policies",
 "mcp_protocol_version": "2025-06-18",
 "resource_type": "mcp-server"
}

You can see our Agentgateway correctly returns the PRM. A client should be able to automatically continue the OAuth/OIDC flow from this.

HOWEVER.

You can see the resource field gets set to "https://ceposta-agw.ngrok.io/entra/mcp",

This causes an issue in MCP inspector (not VS Code or other clients) because MCP inspector uses this field in the Authorization request. It sets the resource parameter based on this value (according to spec):

resource=https://ceposta-agw.ngrok.io/entra/mcp
scope=api%3A%2F%2Fb92d6e60-86ff-4359-b971-04404fe079ec%2Fmcp_access

The problem is, this is not the correct resource in Entra. The correct resource in entra is our api://b92d6e60-86ff-4359-b971-04404fe079ec client_id. If we configure the Agentgateway to return this in our PRM, then that breaks the rest of the OAuth flow, since that’s not the real URL for our MCP resource.

The fix here is to:

Make MCP Inspector more flexible to override the resource sent in the auth flows
Use a verified custom domain the app registration resource URI.

If you are doing this in production, you’d want to user a verified custom domain. Alternatively, in your custom MCP clients, make a way to override the resource parameter when calling the authorize endpoint and use the real client_id like this example:

https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/authorize?
response_type=code&
client_id=9beda151-9370-42f2-a2f7-17933c5c5a7c&
code_challenge=BPfzTvHnOhmbZyB8aIW3sCG69vLmF_hG3aQRvl5C31s&
code_challenge_method=S256&
redirect_uri=http%3A%2F%2Flocalhost%3A6274%2Foauth%2Fcallback%2Fdebug&
state=38158e70909ff3264e129b13b3927b34dc299a7d2dea3b9ee62ca0f181e83db5&
scope=api%3A%2F%2Fb92d6e60-86ff-4359-b971-04404fe079ec%2Fmcp_access&
resource=api%3A%2F%2Fb92d6e60-86ff-4359-b971-04404fe079ec

Wrapping Up

The right pattern for enterprise MCP usage is to tie access to the internal IdP and SSO. All policy should be written against these user IDs (and groups, claims, etc). For more complex auth flows that require cross-identity OAuth ie, like if you need to call GitHub MCP servers, or Databricks, etc each which have their own IdP separate from enterprise IdP, then you can do that also with Agentgateway Enterprise:

Provider Rate Limiting is Not Enough for Enterprise LLM Usage

Sun, 02 Nov 2025 00:00:00 +0000

Hosted LLM providers like OpenAI and Anthropic have rate limiting capabilities centered around requests per minute (RPM) and tokens per minute (TPM). These are specified at the “organization” level and “project” level. API keys are associated with an organization or project, and each call is subject to token or rate limit restrictions. The actual limits are set by the provider, not you. They will depend on what tier you pay for, but generally not directly configurable. For example:

Provider	Model	Typical TPM Limit (per project, tier 1 example)	Notes
OpenAI	GPT-3.5 Turbo	1,000,000 TPM	Popular mid-tier model
OpenAI	GPT-4.1	1,000,000 TPM	Advanced, large context window
OpenAI	GPT-5 Tier 1	500,000 TPM	Raised from 30K TPM recently
Anthropic	Claude 3.5 Sonnet	1,000,000 TPM	Enterprise tier example
Anthropic	Claude 3.0	400,000 TPM	Lower tier example

In essence the providers give you coarse grained, non-configurable controls for rate limiting. Enterprises need far more control here. Enterprises need:

Fine-grained rate limit / quota enforcement
Detailed report/audit/metrics for chargeback/showback
Model/provider failover
Configurable rate limit / metric collection

Providers expect users to figure this out themselves. That’s where something like an LLM gateway (ie, agentgateway) comes into the picture.

In this blog post, you’ll learn how to properly set fine-grained rate limiting, understand cost/usage of Tokens, Model/LLM failover, and observability for agentgateway.

TL;DR video demo

You can see a demo of these features in action here:

What do Enterprises Need?

The hosted LLM providers are trying to protect their service, and rightfully so. They give you fixed buckets for service, and coarse grained limiting that cannot be configured. Enterprises need things like fine-grained rate limit control, attribution metrics, spike arrest, model (and potentially provider) failover, and dashboards/alerting. Most of this should NOT be implemented on the hosted LLM providers (for privacy, PII, and compliance reasons - e.g., sending user IDs, auth context, entitlements, etc), so enterprises are leveraging AI / LLM gateways to do this.

Think of this like cloud providers. You’re essentially “renting infrastructure” from them, and there is a cap/quota/limit that each region has. The same type of concept applies to LLM providers.

Let’s see how agentgateway can be used here.

Fine Grained Rate Limiting, Cost Control, Spike Arrest

Agentgateway can perform rate limiting based on “requests” or “tokens” just like you see from the providers. The time window is configurable, unlike with the providers. You can specify your rate limiting window in terms of seconds, minutes, hours, days, etc.

The key to getting “fine-grained” rate limit is in how agentgateway applies these limits. It can do so on a local or global basis. For example, if we configure rate limits locally with the following:

policies:
 localRateLimit:
 - maxTokens: 1000
 tokensPerFill: 1000
 fillInterval: 1s
 type: tokens

… then each instance of agentgateway will allow 1000 tokens per second. But since we can configure the tokensPerFill we can get sophisticated enough to implement a spike arrest / moving window for our rate limit. For example:

policies:
 localRateLimit:
 - maxTokens: 1000
 tokensPerFill: 100
 fillInterval: 6s
 type: tokens

With this configuration, we could receive a spike of 1000 tokens at any point during our time window but we would refill the rate limit bucket with 100 tokens every 6s. If we had a consistent usage of around 16 tokens per second (100/6) we would never trigger the rate limit. If we get bursts, ie, we could handle them up to a point (ie, 1000 tokens). This type of rate limiting can be applied at the route + matcher level. For example, you can have different rate limit policies on /team-a/openai vs /team-b/openai.

We can also configure a global rate limit policy that would affect all instances of agentgateway. We would need to introduce a rate limit service that uses something like Redis to store its counters. We would then configure the global rate limiter like this on agentgateway:

policies:
 remoteRateLimit:
 domain: "agentgateway"
 host: "${RATELIMIT_HOST:-localhost}:8081"
 descriptors:
 - entries:
 - key: "route"
 value: '"anthropic"'
 type: "tokens"

Global rate limiting works by sending “descriptors” to a rate limiting service. You can think of these descriptors as tags, or metadata about the request that will be used to match on in the rate limit service. For example, in the above, we send a single descriptor called “route” with a value of “anthropic”. We also specify to the rate limit service that this is of type “tokens”.

In the rate limit server, we can configure something like this:

 - key: route
 value: "anthropic"
 rate_limit:
 unit: minute
 requests_per_unit: 5000

What this config does is match on the set of descriptors (in this case, “route” and the value of “anthropic), specify the time window (minute) and the # of requests/tokens per time window. In this case, we’d get 5k tokens per minute for the anthropic route. But like I said, the time window is completely configurable.

This descriptor / matcher is very powerful. We can have hierarchy, match on specific users, IP addresses, organization roles, entitlements, environments, clouds, etc. Here’s a more sophisticated configuration:

policies:
 remoteRateLimit:
 domain: "agentgateway"
 host: "${RATELIMIT_HOST:-localhost}:8081"
 descriptors:
 - entries:
 - key: "route"
 value: '"anthropic"'
 type: "tokens"
 - entries:
 - key: "user_id"
 value: 'jwt.sub'
 type: "tokens"
 - entries:
 - key: "remote_address"
 value: 'string(source.address)'
 type: "tokens"

This will enforce rate limit per route/user/source_ip tuple. This gives extremely fine-grained control over how to administer rate limiting. This can then be used to control LLM spend.

Enriching Call Metrics

Once we have fine-grained control over what clients, applications, users, teams, environments, etc can call which models, and enforce fine-grained usage limits at runtime, we need to track what is actually getting used. Model providers do give some visibility into what gets consumed, but again, the buckets are too coarse grained. The usage limits are tracked per project and per API key. Just like with rate limiting, we’ll need more fine-grained metrics.

Organizations are beginning to think about Agentic Infrastructure like they have been thinking about cloud environments for years utilizing failover, High Availability, Usage (in this case, Tokens), and network connectivity. Because of that, understanding what is going on underneath the hood is crucial for all teams utilizing LLMs.

Agentgateway has a wealth of metrics about LLM usage. The important point here is that the metric dimensions are fully configurable. Let’s take a look. Here are the main metrics that get tracked:

agentgateway_gen_ai_client_token_usage - Tracks the number of tokens used per request to the LLM. Understanding cost and request size distribution.

Example question it answers: “How many tokens are being sent or generated per LLM call?”

agentgateway_gen_ai_server_request_duration - Measures total duration of each generative AI request on the server side. Observing total latency per LLM request (including input processing + generation).

Example question it answers: “How long does an entire LLM call take end-to-end?”

*agentgateway_gen_ai_server_time_to_first_token - Time from when the request starts until the model produces the first output token. Use for Tracking model responsiveness.

Example question it answers: “How fast does the model start responding?”

agentgateway_gen_ai_server_time_per_output_token - Average time it takes the model to generate each token after the first. Use for: Monitoring generation throughput.

Example question it answers: “How fast is the model generating tokens once it starts?”

agentgateway_gen_ai_client_token_usage_sum - Total Tokens

Example question it answers: “How many tokens have been used for a particular LLM?”

agentgateway_downstream_connections_total - Counter of downstream connections (labeled by bind, gateway, listener, protocol)

Example question it answers: “What connections are coming from clients to the Gateway?”

If we annotate the metrics with additional metadata, we can get more depth to what calls are happening, by whom, and how to attribute them: Let’s look at an example:

# HELP agentgateway_gen_ai_client_token_usage Number of tokens used per request.
# TYPE agentgateway_gen_ai_client_token_usage histogram
agentgateway_gen_ai_client_token_usage_bucket{
 model="gpt-4-turbo",
 user_id="auth0|68f7a9cc8fabe01218ebfd58",
 user_email="jane@acme.io",
 team="docs-ai",
 app="knowledge-bot",
 env="prod",
 le="1000"} 58
agentgateway_gen_ai_client_token_usage_bucket{... le="5000"} 72
agentgateway_gen_ai_client_token_usage_sum{...} 168421
agentgateway_gen_ai_client_token_usage_count{...} 93

In this example, Jane from the Docs-AI team used 168K tokens across 93 requests to gpt-4-turbo. By enriching with JWT claims (user_id, team, app, env), you can easily break down token consumption per app, per user, or per environment, giving finance and ops teams visibility into real costs.

Model Failover

While rate limit and attribution makes up a big part of the usage use cases in enterprises to ensure fairness, accounting, and attribution, one aspect that should not get overlooked. Usage is still beholden to the LLM provider and those will have their own usage limits and uptime guarantees. When those limits get exhausted (ie, you hit provider rate limiting / exhaustion of quota) or a particular model has an issue, or the entire provider has out outage, we can’t just sit dead in the water.

Enterprises will need to effectively plan for failover. If traffic is managed through agentgateway, we can explicitly specify failover rules / degradation paths. For example in the following route, we can explicitly plan to failover from gpt-5 to gpt-4o:

- name: failover-openai
 matches:
 - path:
 pathPrefix: /failover/openai
 policies:
 retry:
 attempts: 2 # Retry once (2 total attempts)
 codes: [429] # Retry on 429 errors
 urlRewrite:
 path:
 prefix: ""
 backends:
 - ai:
 groups:
 - providers:
 - name: primary-model
 provider:
 openAI:
 model: "gpt-5"
 backendAuth:
 key: ${OPENAI_API_KEY} 

 - providers:
 - name: secondary-model
 provider:
 openAI:
 model: "gpt-4o"
 backendAuth:
 key: ${OPENAI_API_KEY}

In this case, if gpt-5 has issues or hits provider limits, agentgateway will recognize this (ie, 429s, etc) and failover to the gpt-4o model.

Taking the same approach, there are times where quotas may be hit on a particular AI vendor for the subscription that your organization has or a particular LLM provider is having connectivity issues and failures are occurring. If that happens, you can failover to a different model provider.


 backends:
 - ai:
 groups:
 - providers:
 - name: primary-model
 provider:
 openAI:
 model: "gpt-5"
 backendAuth:
 key: ${OPENAI_API_KEY} 

 # Different provider : Anthropic
 - providers:
 - name: secondary-model
 provider:
 openAI:
 model: "claude-3-5-haiku-latest"
 backendAuth:
 key: ${ANTHROPIC_API_KEY}

It’s important to understand the performance and behavior of the primary model and how/what to failover to. Models each have their nuances when it comes to thinking, tool behavior, accuracy, etc. Some thought needs to be put into failover.

Dashboards / Alerting

With the deep insight and usage metrics from agentgateway (and logging, tracing, etc), we can get detailed dashboards across model usage, cost, latency, and reliability. Because every metric can be annotated with rich metadata — such as user_id, team, application, environment, provider, and model — the resulting dashboards become multidimensional and extremely actionable.

Using a standard Prometheus + Grafana setup (or OpenTelemetry-compatible backend like Datadog, New Relic, etc), enterprises can create visualizations such as:

Usage by Model and Team – Track which models are most heavily used by which teams, and correlate this with cost and output quality.
Token Consumption per Environment / App – See how production vs staging traffic differs in request size and token generation, helping optimize cost and detect runaway workloads.
Latency and Time-to-First-Token Trends – Detect degradations or regressions in model responsiveness across providers.
Rate Limit & Throttling Events – Monitor where clients are being throttled, by what route or policy, to proactively tune quotas.
Failover Behavior – Visualize when and why traffic was shifted to a secondary provider or model, surfacing patterns like repeated rate-limit failures or upstream outages.

These dashboards can easily be sliced and diced by any metric dimension you’ve annotated in agentgateway — for example, breaking down token usage by team, application, or region.

For alerting, since all metrics are exposed in standard formats, you can define rules like:

Trigger alerts when token usage nears a quota threshold per team.
Notify on sustained latency spikes or slow time-to-first-token.
Alert when failover events exceed a certain frequency.
Send cost threshold alerts by model or provider to finance or platform teams.

The combination of agentgateway’s flexible metrics model, Prometheus exposition, and configurable metadata enables both deep operational visibility and automated guardrails. Enterprises can make informed scaling decisions, attribute costs accurately, and quickly detect abnormal behavior across teams, environments, and providers — all without exposing sensitive user data to the hosted LLM providers.

Leaning into Grafana and Prometheus, two popular and prominent open-source monitoring and metrics tools, you can capture the agentgateway metrics within Prometheus and if you want a visual from a monitoring perspective, you can create a Grafana dashboard with those metrics.

You can see the dashboard here

Here’s an example of what the metrics look like from a visual perspective.

Wrapping Up

Enterprises know that managing LLM usage isn’t just about calling an API or relying fully on what the provider has. It’s about governance, control, visibility, and resilience. Agentgateway gives you the building blocks to do this: fine-grained rate limiting, attribution, failover, and metrics/dashboards based on Open Telemetry all without sending sensitive context to hosted LLM providers. If you’re building AI into your enterprise applications, it’s time to treat your LLM traffic like any other critical service layer. Try agentgateway today and see how easy it is to gain control, insight, and reliability over your AI infrastructure.

Solo.io Contributes agentgateway to Linux Foundation to Make AI Agents More Accessible, Capable, and Secure

Mon, 25 Aug 2025 00:00:00 +0000

Today at Open Source Summit Europe, the Linux Foundation accepted agentgateway, a new open source AI-native project created by Solo.io. Agentgateway provides drop-in security, observability, and governance for agent-to-agent and agent-to-tool communication and supports leading interoperable protocols, including Agent2Agent (A2A) and Model Context Protocol (MCP).

Building the Go-To AI Gateway

There are many gateways available today, but most were designed before the rise of AI agents and struggle to support modern AI protocols without major rearchitecture. As a company with deep expertise in Envoy, we initially considered it as the foundation for agentgateway. However, we quickly realized that supporting modern agent protocols like A2A and MCP would require a significant re-architecture of Envoy itself.

To keep pace with rapid innovation in the age of AI, we needed a purposely built solution designed specifically for AI agents. That’s how agentgateway was born.

We didn’t create agentgateway from scratch. Over the past three years, we’ve been building Istio ambient service mesh within the community. A key component of Istio ambient is the zero trust tunnel (ztunnel) — a purpose-built, lightweight proxy designed to handle the secure overlay layer. Today, many adopters run Ambient with ztunnel in production at scale.

We’ve taken the lessons learned from building ztunnel and applied them to agentgateway, enabling us to create the most advanced AI-native proxy in the ecosystem. Agentgateway is the first and only data plane built from the ground up for AI agents, governing and securing communication across agent-to-agent, agent-to-tool and agent-to-LLM interactions.

“The future of software is agentic and that changes everything about how systems connect and communicate. Existing API gateways weren’t designed for the rapidly evolving networking demands of AI and agentic architectures, and they can’t adapt fast enough.” said Idit Levine, CEO of Solo.io. “We built agentgateway from the ground up to handle the protocols, patterns, and scale required for agentic infrastructure - from A2A and MCP to LLM provider APIs and high-volume inferencing. Agentgateway is the connective tissue for the next generation of intelligent systems.”

Enterprise Use Cases Take Off

As enterprises rapidly adopt agentic AI to automate workflows and enhance productivity, they face a growing challenge: ensuring secure, observable, and governed connectivity between AI agents, tools, and large language models.

In real-world enterprise environments:

Agents do not operate in isolation. They interact with each other (agent-to-agent), with internal systems (agent-to-tool), and with external or foundational models (agent-to-LLM).
These interactions are often dynamic, multi-modal, and span organizational and data boundaries.

This creates new vectors for risk and complexity, including:

Security: How do we authenticate, authorize, and audit agent interactions across tools and services?
Governance: How do we enforce policies (e.g., data residency, access control) across autonomous workflows?
Observability: How do we gain visibility into what agents are doing, when, and why?

Agentgateway is designed to tackle these enterprise challenges—security, governance, and observability—at their core.

End User & Community Excitement Abounds

“The future won’t be built by standalone agents, MCP servers or LLMs — it’s shaped by their interconnection and ability to work together seamlessly. To unlock their full potential, we must apply policies, ensure control and maintain clear visibility into their interactions. This is where agentgateway plays a pivotal role — bridging not only agent-to-agent (A2A) communication but also agent-to-MCP servers, filling a critical gap in the ecosystem. I look forward to seeing the project’s continued momentum within the Linux Foundation.”
– John Roese, global chief technology officer & chief AI officer, Dell

“AI agents are rapidly transforming how enterprises work and innovate. To adopt them responsibly at scale, organizations need open and interoperable gateways that provide governance, visibility, and security. Agentgateway delivers the foundation enterprises need. I’m excited to see the community come together to accelerate the open foundation our customers need to scale AI on cloud platforms like CoreWeave.”
– Chen Goldberg, senior vice president of engineering at CoreWeave.

“Agentic AI demands purpose-built infrastructure, not just another software layer. This requires rethinking compute, storage, and data movement from the ground up, so retrofitting legacy systems doesn’t work. The agentgateway project is a solid step toward that future. We’re happy to see it hosted by the Linux Foundation, where open source and community can drive the adoption and longevity that AI infrastructure requires.”
– Jon Alexander, senior vice president, cloud technology, Akamai

“We are excited to welcome the agentgateway project to the Linux Foundation, ensuring that best practices for agentic workflows remain free and open to all. Agentgateway complements emerging specifications like A2A and MCP, and offers scalable, specification-aligned infrastructure for agent communication thereby empowering customers and developers to build robust agentic workflows across platforms.”
– Mitch Connors, CNCF Ambassador (Microsoft)

“I’m excited to see Solo.io donate the agentgateway project to the Linux Foundation. Open sourcing an AI-native connectivity solution that understands MCP and A2A is a big win for the community. It helps us scale AI responsibly, with the security, flexibility, and governance we need in the real world.”
— Rob Hansen, director, digital product engineering and platforms, T-Mobile

“Building reliable AI agents is a challenge, especially when every step involves non-deterministic calls to LLMs, tools, and autonomous agents. Agentgateway’s integration with OpenTelemetry provides a robust foundation for observability, allowing us to treat each request-response pair as an evaluable unit. This capability is crucial for ensuring system-level accuracy and trustworthiness, paving the way for a true ‘AI mesh’ that empowers teams to scale, secure and optimize their AI workflows.”
— Sathish Krishnan, executive director/distinguished engineer, cloud & AI, UBS

“One of the biggest open security problems today is how to do MCP security effectively. While there are a lot of problems in this space that the community doesn’t know how to address, the agentgateway project provides a first step toward addressing some of the important issues with basic role-based access control and visibility of actions to MCP servers. I look forward to seeing how this project adapts and evolves to handle the complex, evolving threats in this space under open source governance.”
– Justin Cappos, professor at New York University and creator of the TUF, Uptane and in-toto projects

“The rapid evolution of the AI landscape demands robust, vendor-neutral infrastructure for how agents communicate with each other and with external tools. Without it, we risk stifling innovation and adoption. The agentgateway project, hosted by the Linux Foundation, is a crucial step in creating that common ground. We are excited to partner with Solo.io and support a community-driven foundation for the future of interoperable AI.”
– Jim Bugwadia, creator, Kyverno and CEO, Nirmata

This is a critical time in our industry as organizations seek to gain compelling benefits from AI agents. The acceptance of agentgateway as an open source project by The Linux Foundation fills a vital need in the ecosystem of AI open standards. The leadership demonstrated by Solo.io enables tech vendors to continue delivering AI innovations at an unprecedented pace."
— Mitch Ashley, vice president and practice lead, software lifecycle engineering, The Futurum Group

The Next Chapter Under the Linux Foundation

The agentgateway project under the neutral governance of the Linux Foundation will ensure that this critical component remains vendor-agnostic and community-driven. This move is designed to accelerate the adoption and development of the agentgateway by providing a full feature purposefully designed for AI agent gateway for open collaboration, intellectual property management, and long-term stewardship.

“The rise of AI agents depends on a strong foundation of open source infrastructure that is built to last. The agentgateway project provides a centralized and secure management layer for AI agent interactions, supporting emerging standards like the Model Context Protocol, " said Jim Zemlin. “We’re pleased to welcome agentgateway to the Linux Foundation where it will benefit from the neutral governance and global community needed to build open, interoperable, secure agent systems for the next generation of AI applications.”

We continue to build the best open agentgateway for AI agents with our partners and others in the space, leveraging a broader set of open standards such as A2A or MCP across topics like:

Agent identity (impersonation and delegation)
Agent naming services
Agent security
Interoperability with various agent frameworks, and more

Join the Agentgateway Community

We welcome anyone passionate about the future of AI agents to join us. Since launching community meetings in July, we’ve seen early participation and contributions from leading organizations including AWS, Microsoft, Red Hat, IBM, Zayo, Shell, Cisco, and Huawei.

To get involved:

Visit the agentgateway project GitHub
Join the conversation on Discord
Attend our weekly community meetings

Let’s build the future of AI agents—together.

MCP Authorization the Easy Way

Tue, 12 Aug 2025 00:00:00 +0000

In June 2025, the MCP community updated the specification to alleviate some of the concerns from the previous version regarding MCP Authorization. However, this update introduces new concerns, especially around enterprise usage. Nevertheless, many public facing MCP clients (Claude, VS Code, etc) do implement the MCP Authorization spec, and many public facing MCP services are expecting this.

In agentgateway, we are trying to make this easier for those building MCP servers. In recent builds, we’ve introduced a way to configure mcpAuthentication which leverages an external OAuth provider, specifically implementing the MCP server side of the MCP Authorization spec for you via configuration.

The MCP specification says the following about the MCP Server’s responsibilities in the authorization spec:

MCP servers MUST use the HTTP header WWW-Authenticate when returning a 401 Unauthorized to indicate the location of the resource server metadata URL as described in RFC9728 Section 5.1 “WWW-Authenticate Response”.

And also…

MCP servers MUST implement the OAuth 2.0 Protected Resource Metadata (RFC9728) specification to indicate the locations of authorization servers. The Protected Resource Metadata document returned by the MCP server MUST include the authorization_servers field containing at least one authorization server.

For example, if you’ve built an MCP server (npx, python, remote, whatever), and you want to handle these spec requirements consistently for your MCP server (and potentially exposing many others), you can use agentgateway to do this.

For example, if this is how you route to your MCP Server:

binds:
- listeners:
 - routes:
 - backends:
 - mcp:
 targets:
 - name: hello-world
 stdio:
 args:
 - 'run'
 - 'src/main.py'
 cmd: 'uv'
 matches:
 - path:
 exact: /hello/mcp
 policies:
 cors:
 allowHeaders:
 - mcp-protocol-version
 - content-type
 allowOrigins:
 - '*'
 port: 3000

Then you can add the new mcpAuthentication configuration to implement this behavior:

 mcpAuthentication:
 issuer: http://localhost:7080/realms/mcp
 jwks:
 url: http://localhost:7080/realms/mcp/protocol/openid-connect/certs
 audience: mcp_proxy
 provider:
 keycloak: {}
 resourceMetadata:
 resource: http://localhost:3000/hello/mcp
 scopesSupported:
 - profile
 - offline_access
 - openid
 bearerMethodsSupported:
 - header
 - body
 - query
 resourceDocumentation: http://localhost:3000/hello/docs
 resourcePolicyUri: http://localhost:3000/hello/policies

There are a couple important points to make here. When we use this configuration, we automatically get a spec-compliant implementation for MCP Auth. But, depending on the OAuth provider we use, it may need some massaging. For example, Keycloak has some challenges with Dynamic Client Registration (ie, registration endpoint, CORS, etc). agentgateway can automatically wrap this and expose it in a way that’s spec compliant. That’s the point of the provider: keycloak {} configuration.

Out of the box provider: auth0 {} is also supported. If your provider correctly supports all of the OAuth RFC / specs, then you can omit provider: completely.

Take a look at this demo video to see how it all works:

Announcing A2A, MCP, and Kubernetes Gateway API support

Mon, 14 Jul 2025 00:00:00 +0000

Today, we’re excited to share the next major milestone: agentgateway is now a full-featured, AI-native gateway that combines deep MCP and A2A protocol awareness, robust traffic policy controls, inference gateway support, Kubernetes Gateway API support, and unified access to major LLMs, all purpose-built with Rust for real-world agentic systems.

Back when we first introduced agentgateway, it was designed to fill a critical gap in the AI stack: enabling structured, secure, and scalable communication between agents, tools, and LLMs using protocols like MCP and A2A. Solo.io wrote about this in a blog post where we explained why traditional API gateways fall short in agentic environments. Since then, the project has grown tremendously.

Today, we’re excited to share the next major milestone: Agentgateway is now a full-featured, AI-native gateway that combines deep MCP and A2A protocol awareness, robust traffic policy controls, inference gateway support, Kubernetes Gateway API support, and unified access to major LLMs, all purpose-built with Rust for real-world agentic systems.

And notably, agentgateway now fully supports:

The latest MCP spec (2025-06-18), including latest authorization changes
The v0.2.x release of A2A

Let’s take a closer look at the updated capabilities.

Deep Protocol Awareness: MCP and A2A

Agentgateway continues to deepen its native support for the emerging Model Context Protocol (MCP) and Agent-to-Agent Protocol (A2A) with updated support including:

Protocol-aware routing, telemetry, and tracing
Support for MCP server/tool aggregation (virtualized MCP servers)
Updated MCP Authorization support per the June 2025 spec
Native authorization policy engine using Cedar for fine-grained authorizations
Support for exposing local stdio-based MCP servers as remote targets

Agentgateway can be used to implement authentication and authorization for your MCP servers with minimal set up. We will keep agentgateway updated as these specs continue to evolve.

Kubernetes Gateway API Support

Agentgateway now implements the Kubernetes Gateway API with support for HTTPRoute, GRPCRoute, TCPRoute, and TLSRoute. This isn’t just basic support, we’ve implemented all core and extended features, and most experimental ones too. This means you can:

Match traffic based on headers, paths, and query parameters
Apply CORS, redirects, header rewrites, request mirroring
Configure timeouts, retries, and direct responses

For platform teams already using Kubernetes-native tooling, this makes agentgateway a seamless drop-in with advanced L7 control. It’s a critical step toward integrating agentic workloads into the broader platform ecosystem.

Fine-Grained Traffic Policy Controls

In real enterprise deployments, security and reliability aren’t optional. That’s why agentgateway now includes advanced policy features:

Local and remote rate limiting
JWT authentication and external auth, (ie, extAuthz) hooks
Upstream auth (cloud identity, TLS, etc)
Full OpenTelemetry support for metrics, logs, and distributed tracing

These controls allow you to run production-grade agent infrastructure that meets enterprise security and observability requirements.

LLM Gateway

Agentgateway can now be used as an “AI Gateway” and route traffic directly to LLMs (OpenAI, Anthropic, Gemini, Bedrock, etc). Top use cases here are unified LLM API, prompt guarding, and resilience (failover, rate limiting, etc).

Agentgateway now provides a unified OpenAI-compatible API across:

OpenAI / Azure OpenAI
Anthropic
Google Gemini
Amazon Bedrock
Google Vertex

This enables users to seamlessly move between providers without changes to their application, even dynamically based on the health or performance of a specific provider.

Agentgateway can also be used to implement prompt guarding to scan/filter for sensitive information or full on direct prompt injection attacks:

Regex-based filters to block prompts with known unsafe patterns
Optional webhook integration to run pre-flight validation using custom logic

Combined with the fine-grained traffic policy controls, agentgateway can act as a powerful LLM Gateway.

Inference Routing

If you are running your own models on self-hosted GPU infrastructure, agentgateway now implements the Inference Gateway extensions for more accurate, efficient prompt routing. Using the InferencePool API, you can route based on:

Prompt criticality
GPU and KV cache utilization
Work queue / waiting queue depth
Lora adapters

This gives AI platform operators more efficient and cost-conscious routing. This can also form the foundation for deeper optimizations like llm-d.

Get Involved

We’re thrilled to see the community interest grow around this project. If you’re building agent infrastructure, dealing with LLM routing, or trying to enforce policy in AI-native environments, we’d love your feedback and contributions:

Agentgateway has evolved far beyond just a proxy. It’s becoming the control point for secure, scalable agentic systems. We’re just getting started.