LLM (OpenAI)

Verified

Configure the agentgateway binary to route requests to the OpenAI chat completions API.

Before you begin

Install the agentgateway binary.
To install the latest release:
curl -sL https://agentgateway.dev/install | bash
To install a specific version, pass the --version flag. Use any release tag from the agentgateway releases page, such as 1.1.0.
curl -sL https://agentgateway.dev/install | bash -s -- --version 1.1.0
To install the nightly build for development and testing:
1. Go to the nightly release in GitHub Actions and click the release that you want to use.
2. From the URL, get the release number, such as 24873456345 in https://github.com/agentgateway/agentgateway/actions/runs/24873456345.
3. Using the gh CLI, download the release for your OS. The following example uses macOS.
  gh run download 24873456345 -R agentgateway/agentgateway -n release-binary-mac
4. Make the binary file executable and move it to your binary location, such as in the following example.
  chmod +x agentgateway sudo mv agentgateway /usr/local/bin/agentgateway
5. Verify that you have the nightly release.
  agentgateway --version
  Example output:
  { "version": "0.0.0-alpha.813d7d0", "git_revision": "813d7d0ab4757db7c8ed5a639bc63c0bb20ac116", "rust_version": "1.95.0", "build_profile": "release", "build_target": "aarch64-apple-darwin" }

Get an OpenAI API key.

Steps

Route to an OpenAI backend through agentgateway.

Step 1: Set your API key

Store your OpenAI API key in an environment variable so agentgateway can authenticate to the API.

export OPENAI_API_KEY='<your-api-key>'

Step 2: Create the configuration

Create a config.yaml that defines an LLM model for OpenAI. This configuration uses the simplified LLM format to route traffic to the OpenAI backend.

cat > config.yaml << 'EOF'
# yaml-language-server: $schema=https://agentgateway.dev/schema/config

llm:
  models:
  - name: gpt-3.5-turbo
    provider: openAI
    params:
      model: gpt-3.5-turbo
      apiKey: "$OPENAI_API_KEY"
EOF

Step 3: Start agentgateway

Run agentgateway with the config file.

agentgateway -f config.yaml

Example output:

info  state_manager  loaded config from File("config.yaml")
info  app            serving UI at http://localhost:15000/ui
info  proxy::gateway started bind  bind="bind/4000"

Step 4: Send a chat completion request

From another terminal, send a request to the chat completions endpoint.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }' | jq .

Example output (abbreviated):

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help you today?"
      }
    }
  ]
}

Next steps

Check out more guides related to LLM consumption with agentgateway.

Control spending

Control spending by setting rate limits for your LLM requests.

LLM observability

View metrics, traces, and logs for LLM traffic.

OpenAI provider reference

Optional model override, multiple routes, passthrough, and Codex connection.

MCP servers

Was this page helpful?