OpenAI SDK

Use the OpenAI Python and Node.js SDKs with agentgateway running in Kubernetes

Use the OpenAI Python or Node.js SDK to send requests through agentgateway deployed in Kubernetes.

Before you begin

1. Set up an agentgateway proxy.
2. Set up access to the OpenAI LLM provider.

Get the gateway URL

export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo "Gateway address: $INGRESS_GW_ADDRESS"

export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "Gateway address: $INGRESS_GW_ADDRESS"

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80

Python

1. Install the OpenAI SDK in your Python project.

   pip install openai

2. Create and run the following script to send a request through agentgateway. Replace <route-path> with the path from your HTTPRoute configuration (for example, /openai).

   Do not include /v1 in the base_url — the OpenAI SDK appends it automatically.

   import os
   from openai import OpenAI
   
   gateway_address = os.environ["INGRESS_GW_ADDRESS"]
   
   client = OpenAI(
       base_url=f"http://{gateway_address}/<route-path>",
       api_key="anything",  # placeholder if gateway has no auth
   )
   
   response = client.chat.completions.create(
       model="gpt-4o-mini",
       messages=[{"role": "user", "content": "Hello from Kubernetes!"}],
   )
   print(response.choices[0].message.content)

Node.js

1. Install the OpenAI SDK in your Node.js project.

   npm install openai

2. Create and run the following script to send a request through agentgateway. Replace <route-path> with the path from your HTTPRoute configuration (for example, /openai).

   Do not include /v1 in the baseURL — the OpenAI SDK appends it automatically.

   import OpenAI from "openai";
   
   const gatewayAddress = process.env.INGRESS_GW_ADDRESS;
   
   const client = new OpenAI({
     baseURL: `http://${gatewayAddress}/<route-path>`,
     apiKey: "anything",
   });
   
   const response = await client.chat.completions.create({
     model: "gpt-4o-mini",
     messages: [{ role: "user", content: "Hello from Kubernetes!" }],
   });
   console.log(response.choices[0].message.content);