For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
OpenAI SDK
Use the OpenAI Python and Node.js SDKs with agentgateway running in Kubernetes
Call configured LLM providers from your application code through agentgateway in Kubernetes with the OpenAI SDK for Python, Node.js, or Go.
Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at the gateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the configured provider and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.
Before you begin
Get the gateway URL
Tip
Kind cluster? Kind does not support LoadBalancer services by default. To use this option with a Kind cluster, install and run cloud-provider-kind.
The following command reads the LoadBalancer IP address or hostname, whichever your cloud provider assigns.
export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
-o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo "Gateway address: $INGRESS_GW_ADDRESS"If you use port-forwarding, keep it running and set the address in the terminal where you run the SDK examples:
export INGRESS_GW_ADDRESS=localhost:8080Set the base URL
End the SDK base URL with /v1, such as http://$INGRESS_GW_ADDRESS/v1. The SDK appends the API path, such as /chat/completions, to the base URL.
If your HTTPRoute matches a path prefix, such as /openai, put the prefix before /v1: http://$INGRESS_GW_ADDRESS/openai/v1.
Python
Install the OpenAI SDK in your Python project.
pip install openaiCreate and run the following script to send a request through agentgateway.
import os from openai import OpenAI gateway_address = os.environ["INGRESS_GW_ADDRESS"] client = OpenAI( base_url=f"http://{gateway_address}/v1", api_key="anything", # placeholder if gateway has no auth ) response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello from Kubernetes!"}], ) print(response.choices[0].message.content)
Node.js
Install the OpenAI SDK in your Node.js project.
npm install openaiCreate and run the following script to send a request through agentgateway.
import OpenAI from "openai"; const gatewayAddress = process.env.INGRESS_GW_ADDRESS; const client = new OpenAI({ baseURL: `http://${gatewayAddress}/v1`, apiKey: "anything", }); const response = await client.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello from Kubernetes!" }], }); console.log(response.choices[0].message.content);
Go
Add the OpenAI Go SDK to your module.
go get github.com/openai/openai-go/v3Create and run the following program to send a request through agentgateway.
package main import ( "context" "fmt" "os" "github.com/openai/openai-go/v3" "github.com/openai/openai-go/v3/option" ) func main() { client := openai.NewClient( option.WithBaseURL("http://"+os.Getenv("INGRESS_GW_ADDRESS")+"/v1"), option.WithAPIKey("anything"), ) resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{ Model: "gpt-4o-mini", Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello from Kubernetes!")}, }) if err != nil { panic(err) } fmt.Println(resp.Choices[0].Message.Content) }
Authenticate with a JWT
Configure an AgentgatewayPolicy with on the gateway. Use mode: Strict to require a valid token on every request.
Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.
Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.
import os
from openai import OpenAI
jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1",
api_key=jwt_token,
)This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the Kubernetes secrets referenced by your backend or model resources. Obtain a fresh token when it expires and update the SDK client before making more requests.