Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

OpenAI SDK

Page as Markdown

Use the OpenAI Python and Node.js SDKs with agentgateway running in Kubernetes

Call configured LLM providers from your application code through agentgateway in Kubernetes with the OpenAI SDK for Python, Node.js, or Go.

Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at the gateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the configured provider and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.

Before you begin

  1. Set up an agentgateway proxy.
  2. Set up access to the OpenAI LLM provider.

Get the gateway URL

Tip

Kind cluster? Kind does not support LoadBalancer services by default. To use this option with a Kind cluster, install and run cloud-provider-kind.

The following command reads the LoadBalancer IP address or hostname, whichever your cloud provider assigns.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
  -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Gateway address: $INGRESS_GW_ADDRESS"

If you use port-forwarding, keep it running and set the address in the terminal where you run the SDK examples:

export INGRESS_GW_ADDRESS=localhost:8080

Set the base URL

End the SDK base URL with /v1, such as http://$INGRESS_GW_ADDRESS/v1. The SDK appends the API path, such as /chat/completions, to the base URL.

If your HTTPRoute matches a path prefix, such as /openai, put the prefix before /v1: http://$INGRESS_GW_ADDRESS/openai/v1.

Python

  1. Install the OpenAI SDK in your Python project.

    pip install openai
  2. Create and run the following script to send a request through agentgateway.

    import os
    from openai import OpenAI
    
    gateway_address = os.environ["INGRESS_GW_ADDRESS"]
    
    client = OpenAI(
        base_url=f"http://{gateway_address}/v1",
        api_key="anything",  # placeholder if gateway has no auth
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello from Kubernetes!"}],
    )
    print(response.choices[0].message.content)

Node.js

  1. Install the OpenAI SDK in your Node.js project.

    npm install openai
  2. Create and run the following script to send a request through agentgateway.

    import OpenAI from "openai";
    
    const gatewayAddress = process.env.INGRESS_GW_ADDRESS;
    
    const client = new OpenAI({
      baseURL: `http://${gatewayAddress}/v1`,
      apiKey: "anything",
    });
    
    const response = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: "Hello from Kubernetes!" }],
    });
    console.log(response.choices[0].message.content);

Go

  1. Add the OpenAI Go SDK to your module.

    go get github.com/openai/openai-go/v3
  2. Create and run the following program to send a request through agentgateway.

    package main
    
    import (
    	"context"
    	"fmt"
    	"os"
    
    	"github.com/openai/openai-go/v3"
    	"github.com/openai/openai-go/v3/option"
    )
    
    func main() {
    	client := openai.NewClient(
    		option.WithBaseURL("http://"+os.Getenv("INGRESS_GW_ADDRESS")+"/v1"),
    		option.WithAPIKey("anything"),
    	)
    
    	resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
    		Model:    "gpt-4o-mini",
    		Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello from Kubernetes!")},
    	})
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println(resp.Choices[0].Message.Content)
    }

Authenticate with a JWT

Configure an AgentgatewayPolicy with on the gateway. Use mode: Strict to require a valid token on every request.

Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.

Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.

import os
from openai import OpenAI

jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
    base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1",
    api_key=jwt_token,
)

This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the Kubernetes secrets referenced by your backend or model resources. Obtain a fresh token when it expires and update the SDK client before making more requests.

Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.