For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
LLM consumption
Overview of LLM provider integration and consumption patterns
Consume LLM services by setting up AI backends for your LLM providers.
About
Overview of supported LLM providers and their capabilities
Providers
Virtual key management
Issue API keys with per-key token budgets and cost tracking (also known as virtual keys).
Manage API keys
Manage API keys for LLM provider authentication.
Content-based routing
Route requests to different LLM backends based on request body content, such as the requested model …
Control spend
Control cost with token budgets and spend limits to prevent unexpected bills and LLM misuse.
OpenAI Realtime
Proxy OpenAI Realtime API WebSocket traffic and track token usage.
Prompt templates
Use static and dynamic prompt templates to customize LLM requests.
Transform requests
Dynamically compute and set LLM request fields using CEL expressions.
Guardrails
Protect LLM interactions with prompt guards that evaluate and filter requests and responses for …
Observe traffic
Get prompt logging, cost tracking, and an audit trail for LLM traffic.
Routing-based configuration for LLMs
Use traditional HTTP routing configuration for advanced use cases like path-based routing and …
