Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

OpenAI SDK

Page as Markdown

Use OpenAI SDK (Python and Node.js) with agentgateway

Call configured LLM providers from your application code through agentgateway with the OpenAI SDK for Python, Node.js, or Go.

Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at agentgateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the provider that matches the model field and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.

Before you begin

Install the agentgateway binary.

Tip

If you manage models and virtual API keys in the standalone UI, Client Setup can generate the connection settings or snippet for this client. Review the gateway URL, select a model and key, choose the client from the Integration dropdown, and copy the recipe. Each recipe uses only the values that its client supports.

Client Setup generates client-side values from existing configuration. It does not create a route, model, authentication policy, virtual key, or provider credential. Follow the steps in this guide to configure those prerequisites or to set up the client manually.

Example agentgateway configuration

Python

  1. Install the OpenAI SDK in your Python project.

    pip install openai
  2. Create and run the following script to send a request through agentgateway.

    from openai import OpenAI
    
    client = OpenAI(
        base_url="http://localhost:3000/v1",
        api_key="anything",  # placeholder if gateway has no auth
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

You can also configure the SDK using environment variables.

export OPENAI_BASE_URL=http://localhost:3000/v1
export OPENAI_API_KEY=anything

Then initialize the client without arguments.

from openai import OpenAI

client = OpenAI()  # picks up OPENAI_BASE_URL and OPENAI_API_KEY from env

Node.js

  1. Install the OpenAI SDK in your Node.js project.

    npm install openai
  2. Create and run the following script to send a request through agentgateway.

    import OpenAI from "openai";
    
    const client = new OpenAI({
      baseURL: "http://localhost:3000/v1",
      apiKey: "anything",
    });
    
    const response = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: "Hello!" }],
    });
    console.log(response.choices[0].message.content);

Go

  1. Add the OpenAI Go SDK to your module.

    go get github.com/openai/openai-go/v3
  2. Create and run the following program to send a request through agentgateway.

    package main
    
    import (
    	"context"
    	"fmt"
    
    	"github.com/openai/openai-go/v3"
    	"github.com/openai/openai-go/v3/option"
    )
    
    func main() {
    	client := openai.NewClient(
    		option.WithBaseURL("http://localhost:3000/v1"),
    		option.WithAPIKey("anything"),
    	)
    
    	resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
    		Model:    "gpt-4o-mini",
    		Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello!")},
    	})
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println(resp.Choices[0].Message.Content)
    }

Authenticate with a JWT

Configure a on agentgateway. Use mode: strict to require a valid token on every request.

Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.

Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.

import os
from openai import OpenAI

jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key=jwt_token,
)

This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the agentgateway configuration. Obtain a fresh token when it expires and update the SDK client before making more requests.

Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.