Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

SDKs (Python, Node.js, Go)

Page as Markdown

Use the OpenAI SDK in Python, Node.js, or Go to call configured LLM providers through agentgateway, and migrate from the LiteLLM SDK.

Call configured LLM providers from your application code through agentgateway with the OpenAI SDK for Python, Node.js, or Go.

Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at agentgateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the provider that matches the model field and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.

Before you begin

Install the agentgateway binary.

Tip

If you manage models and virtual API keys in the standalone UI, Client Setup can generate the connection settings or snippet for this client. Review the gateway URL, select a model and key, choose the client from the Integration dropdown, and copy the recipe. Each recipe uses only the values that its client supports.

Client Setup generates client-side values from existing configuration. It does not create a route, model, authentication policy, virtual key, or provider credential. Follow the steps in this guide to configure those prerequisites or to set up the client manually.

Example agentgateway configuration

You can configure agentgateway in the UI or in your configuration file.

To create the same configuration in the UI, follow these steps.

  1. Set your OpenAI API key, create a configuration file with an empty llm section, and start agentgateway. The UI checks that the OPENAI_API_KEY environment variable exists when you save the model.

    export OPENAI_API_KEY='<your-api-key>'
    cat > config.yaml <<'EOF'
    llm:
      models: []
    EOF
    agentgateway -f config.yaml
  2. Add the model.

    1. Open LLM > Models, such as http://localhost:15000/ui/llm/models, and click Add model.

    2. From the Provider dropdown, select OpenAI.

    3. In the Incoming model match field, replace the default openai/* with *.

    4. For the Provider API key, select Env var, and enter OPENAI_API_KEY.

      Add model drawer with the OpenAI provider, * as the incoming model match, and the provider API key read from the OPENAI_API_KEY environment variable
      Add model drawer with the OpenAI provider, * as the incoming model match, and the provider API key read from the OPENAI_API_KEY environment variable
    5. Click Save model.

  3. Serve LLM traffic on port 3000.

    1. Open the Gateway Overview page, such as http://localhost:15000/ui/.

    2. On the LLM row, click Settings.

    3. Enter 3000 for the Port, and click Save settings.

      LLM Settings drawer with the Gateway binding port set to 3000 and a Save settings button
      LLM Settings drawer with the Gateway binding port set to 3000 and a Save settings button

The UI writes the same deprecated llm.port field as the Config file tab, plus an empty llm.gateways list. If your configuration defines gateways, the Settings drawer shows a Gateway dropdown instead of Port, and binds LLM traffic to the gateway that you select. The UI also writes the provider as openai instead of openAI. Agentgateway accepts both values.

Whether the UI can save depends on your storage mode. For more information, see Save changes from the UI.

Python

  1. Install the OpenAI SDK in your Python project.

    pip install openai
  2. Create and run the following script to send a request through agentgateway.

    from openai import OpenAI
    
    client = OpenAI(
        base_url="http://localhost:3000/v1",
        api_key="anything",  # placeholder if gateway has no auth
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

You can also configure the SDK using environment variables.

export OPENAI_BASE_URL=http://localhost:3000/v1
export OPENAI_API_KEY=anything

Then initialize the client without arguments.

from openai import OpenAI

client = OpenAI()  # picks up OPENAI_BASE_URL and OPENAI_API_KEY from env

Node.js

  1. Install the OpenAI SDK in your Node.js project.

    npm install openai
  2. Create and run the following script to send a request through agentgateway.

    import OpenAI from "openai";
    
    const client = new OpenAI({
      baseURL: "http://localhost:3000/v1",
      apiKey: "anything",
    });
    
    const response = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: "Hello!" }],
    });
    console.log(response.choices[0].message.content);

Go

  1. Add the OpenAI Go SDK to your module.

    go get github.com/openai/openai-go/v3
  2. Create and run the following program to send a request through agentgateway.

    package main
    
    import (
    	"context"
    	"fmt"
    
    	"github.com/openai/openai-go/v3"
    	"github.com/openai/openai-go/v3/option"
    )
    
    func main() {
    	client := openai.NewClient(
    		option.WithBaseURL("http://localhost:3000/v1"),
    		option.WithAPIKey("anything"),
    	)
    
    	resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
    		Model:    "gpt-4o-mini",
    		Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello!")},
    	})
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println(resp.Choices[0].Message.Content)
    }

Call more than one provider

Add a model entry for each provider with a provider prefix. The SDK code stays the same; only the model value changes.

In the following agentgateway configuration file, set OPENAI_API_KEY and ANTHROPIC_API_KEY to your actual provider API keys. The SDK examples use placeholder values for the API keys.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  port: 3000
  models:
  - name: "openai/*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      model: llmRequest.model.stripPrefix("openai/")
  - name: "anthropic/*"
    provider: anthropic
    params:
      apiKey: "$ANTHROPIC_API_KEY"
    transformation:
      model: llmRequest.model.stripPrefix("anthropic/")
messages = [{"role": "user", "content": "Hello!"}]
client.chat.completions.create(model="openai/gpt-4o-mini", messages=messages)
client.chat.completions.create(model="anthropic/claude-sonnet-4-5", messages=messages)

Agentgateway strips the prefix and sends the request to the matching provider. For other ways to name models, see Multiple LLM providers.

Migrate from the LiteLLM SDK

The example configuration in the previous section accepts openai/gpt-4o-mini and anthropic/claude-sonnet-4-5, so you can keep those LiteLLM model strings. Other model strings need matching gateway configuration. This example migrates Chat Completions calls. Review provider-specific parameters and other LiteLLM features separately. Replace the litellm call with the OpenAI SDK and move provider API keys from your application to the agentgateway configuration in Call more than one provider.

Before, with LiteLLM:

from litellm import completion

response = completion(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

After, with agentgateway:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:3000/v1", api_key="anything")

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

For asynchronous calls, replace acompletion with AsyncOpenAI:

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI(base_url="http://localhost:3000/v1", api_key="anything")

async def main():
    response = await async_client.chat.completions.create(
        model="anthropic/claude-sonnet-4-5",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())
LiteLLM SDKOpenAI SDK with agentgateway
completion(...)client.chat.completions.create(...)
await acompletion(...)await async_client.chat.completions.create(...)
stream=Truestream=True
embedding(...)client.embeddings.create(...). See Embeddings.
num_retries=3OpenAI(..., max_retries=3) on the client, or a retry policy on agentgateway
Router(..., fallbacks=[...])A virtual model with failover routing. The client sends the virtual model name.
completion_cost(...)Agentgateway prices each request from its model cost catalog and records the USD cost in access logs, traces, and metrics.
Provider keys, such as ANTHROPIC_API_KEY, in the application environmentparams.apiKey on each model entry in the agentgateway configuration

With LiteLLM fallbacks, the failed request moves to the next model. With agentgateway failover, the request that triggers eviction returns an error, and later requests go to the next priority group. The OpenAI Python SDK retries that error twice by default (max_retries=2), and the retry reaches a model in that group. If you set max_retries=0, configure a retry policy on agentgateway instead.

If you also run the LiteLLM proxy, convert its configuration with Migrate from LiteLLM.

Authenticate with a JWT

Configure a jwtAuth policy on agentgateway. Use mode: strict to require a valid token on every request.

Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.

Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.

import os
from openai import OpenAI

jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key=jwt_token,
)

This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the agentgateway configuration. Obtain a fresh token when it expires and update the SDK client before making more requests.

Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.