OpenAI-compatible providers

OpenAI-compatible providers

Configure any LLM provider that provides OpenAI-compatible endpoints with agentgateway.

Configuration

Review the following example configuration. The example integrates with Cohere AI. For a different provider, consult their documentation to find the provider-specific details.

binds:
- port: 3000
  listeners:
  - routes:
    - policies:
        urlRewrite:
          authority:
            full: api.cohere.ai
          path:
            full: "/compatibility/v1/chat/completions"
        backendTLS: {}
        backendAuth:
          key: $COHERE_API_KEY
      backends:
      - ai:
          name: cohere
          hostOverride: api.cohere.ai:443
          provider:
            openAI:
              model: command
Review the following example configuration.
Setting Description
urlRewrite Configure a policy to rewrite the URL of the upstream requests to match your LLM provider.
authority Set the default hostname authority to forward incoming requests.
path Rewrite the path to the appropriate LLM provider endpoint. This setting is optional if requests on the provider hostname are already sent on this path.
backendTLS Optionally configure a policy to use TLS when connecting to the LLM provider.
backendAuth Optionally configure a policy to attach an API key that authenticate to the LLM provider to outgoing requests. If you do not include an API key, each request must authenticate per the LLM provider requirements.
ai.name The name of the LLM provider for this AI backend.
ai.hostOverride Override the hostname. If not set, the hostname defaults to OpenAI (api.openai.com). This setting is optional if the hostname is already set in the URL rewrite policy’s authority setting.
ai.provider.openAI.model Optionally set the model to use for requests. If not set, the request must include the model to use.