For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
SDKs (Python, Node.js, Go)
Use the OpenAI SDK in Python, Node.js, or Go to call configured LLM providers through agentgateway, and migrate from the LiteLLM SDK.
Call configured LLM providers from your application code through agentgateway with the OpenAI SDK for Python, Node.js, or Go.
Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at agentgateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the provider that matches the model field and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.
Before you begin
Install theagentgateway binary.Tip
If you manage models and virtual API keys in the standalone UI, Client Setup can generate the connection settings or snippet for this client. Review the gateway URL, select a model and key, choose the client from the Integration dropdown, and copy the recipe. Each recipe uses only the values that its client supports.
Client Setup generates client-side values from existing configuration. It does not create a route, model, authentication policy, virtual key, or provider credential. Follow the steps in this guide to configure those prerequisites or to set up the client manually.
Example agentgateway configuration
You can configure agentgateway in the UI or in your configuration file.
To create the same configuration in the UI, follow these steps.
Set your OpenAI API key, create a configuration file with an empty
llmsection, and start agentgateway. The UI checks that theOPENAI_API_KEYenvironment variable exists when you save the model.export OPENAI_API_KEY='<your-api-key>' cat > config.yaml <<'EOF' llm: models: [] EOF agentgateway -f config.yamlAdd the model.
Open LLM > Models, such as http://localhost:15000/ui/llm/models, and click Add model.
From the Provider dropdown, select OpenAI.
In the Incoming model match field, replace the default
openai/*with*.For the Provider API key, select Env var, and enter
OPENAI_API_KEY.

Click Save model.
Serve LLM traffic on port
3000.Open the Gateway Overview page, such as http://localhost:15000/ui/.
On the LLM row, click Settings.
Enter
3000for the Port, and click Save settings.

The UI writes the same deprecated llm.port field as the Config file tab, plus an empty llm.gateways list. If your configuration defines gateways, the Settings drawer shows a Gateway dropdown instead of Port, and binds LLM traffic to the gateway that you select. The UI also writes the provider as openai instead of openAI. Agentgateway accepts both values.
Whether the UI can save depends on your storage mode. For more information, see Save changes from the UI.
Python
Install the OpenAI SDK in your Python project.
pip install openaiCreate and run the following script to send a request through agentgateway.
from openai import OpenAI client = OpenAI( base_url="http://localhost:3000/v1", api_key="anything", # placeholder if gateway has no auth ) response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content)
You can also configure the SDK using environment variables.
export OPENAI_BASE_URL=http://localhost:3000/v1
export OPENAI_API_KEY=anythingThen initialize the client without arguments.
from openai import OpenAI
client = OpenAI() # picks up OPENAI_BASE_URL and OPENAI_API_KEY from envNode.js
Install the OpenAI SDK in your Node.js project.
npm install openaiCreate and run the following script to send a request through agentgateway.
import OpenAI from "openai"; const client = new OpenAI({ baseURL: "http://localhost:3000/v1", apiKey: "anything", }); const response = await client.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello!" }], }); console.log(response.choices[0].message.content);
Go
Add the OpenAI Go SDK to your module.
go get github.com/openai/openai-go/v3Create and run the following program to send a request through agentgateway.
package main import ( "context" "fmt" "github.com/openai/openai-go/v3" "github.com/openai/openai-go/v3/option" ) func main() { client := openai.NewClient( option.WithBaseURL("http://localhost:3000/v1"), option.WithAPIKey("anything"), ) resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{ Model: "gpt-4o-mini", Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello!")}, }) if err != nil { panic(err) } fmt.Println(resp.Choices[0].Message.Content) }
Call more than one provider
Add a model entry for each provider with a provider prefix. The SDK code stays the same; only the model value changes.
In the following agentgateway configuration file, set OPENAI_API_KEY and ANTHROPIC_API_KEY to your actual provider API keys. The SDK examples use placeholder values for the API keys.
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
port: 3000
models:
- name: "openai/*"
provider: openAI
params:
apiKey: "$OPENAI_API_KEY"
transformation:
model: llmRequest.model.stripPrefix("openai/")
- name: "anthropic/*"
provider: anthropic
params:
apiKey: "$ANTHROPIC_API_KEY"
transformation:
model: llmRequest.model.stripPrefix("anthropic/")messages = [{"role": "user", "content": "Hello!"}]
client.chat.completions.create(model="openai/gpt-4o-mini", messages=messages)
client.chat.completions.create(model="anthropic/claude-sonnet-4-5", messages=messages)Agentgateway strips the prefix and sends the request to the matching provider. For other ways to name models, see Multiple LLM providers.
Migrate from the LiteLLM SDK
The example configuration in the previous section accepts openai/gpt-4o-mini and anthropic/claude-sonnet-4-5, so you can keep those LiteLLM model strings. Other model strings need matching gateway configuration. This example migrates Chat Completions calls. Review provider-specific parameters and other LiteLLM features separately. Replace the litellm call with the OpenAI SDK and move provider API keys from your application to the agentgateway configuration in Call more than one provider.
Before, with LiteLLM:
from litellm import completion
response = completion(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)After, with agentgateway:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:3000/v1", api_key="anything")
response = client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)For asynchronous calls, replace acompletion with AsyncOpenAI:
import asyncio
from openai import AsyncOpenAI
async_client = AsyncOpenAI(base_url="http://localhost:3000/v1", api_key="anything")
async def main():
response = await async_client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
asyncio.run(main())| LiteLLM SDK | OpenAI SDK with agentgateway |
|---|---|
completion(...) | client.chat.completions.create(...) |
await acompletion(...) | await async_client.chat.completions.create(...) |
stream=True | stream=True |
embedding(...) | client.embeddings.create(...). See Embeddings. |
num_retries=3 | OpenAI(..., max_retries=3) on the client, or a retry policy on agentgateway |
Router(..., fallbacks=[...]) | A virtual model with failover routing. The client sends the virtual model name. |
completion_cost(...) | Agentgateway prices each request from its model cost catalog and records the USD cost in access logs, traces, and metrics. |
Provider keys, such as ANTHROPIC_API_KEY, in the application environment | params.apiKey on each model entry in the agentgateway configuration |
With LiteLLM fallbacks, the failed request moves to the next model. With agentgateway failover, the request that triggers eviction returns an error, and later requests go to the next priority group. The OpenAI Python SDK retries that error twice by default (max_retries=2), and the retry reaches a model in that group. If you set max_retries=0, configure a retry policy on agentgateway instead.
If you also run the LiteLLM proxy, convert its configuration with Migrate from LiteLLM.
Authenticate with a JWT
Configure a jwtAuth policy on agentgateway. Use mode: strict to require a valid token on every request.
Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.
Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.
import os
from openai import OpenAI
jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
base_url="http://localhost:3000/v1",
api_key=jwt_token,
)This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the agentgateway configuration. Obtain a fresh token when it expires and update the SDK client before making more requests.