For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
Ollama
Configure agentgateway to route LLM traffic to Ollama for local model inference
Configure Ollama to serve local models through agentgateway. Agentgateway 1.3 includes the first-class ollama provider and automatically uses http://localhost:11434/v1 unless you override it.
Before you begin
- Install the
agentgatewaybinary. Install Ollama.
Make sure that you have at least one model pulled locally.
ollama listIf not, pull a model.
ollama pull llama3.2
Configure agentgateway
Test the configuration
Send a request to verify that agentgateway routes to Ollama.
Troubleshooting
Connection refused
What’s happening:
Requests to agentgateway return a 503 response or a connection refused error.
Why it’s happening:
Ollama is not running, is listening on a different address, or params.baseUrl points to the wrong endpoint.
How to fix it:
Verify Ollama is reachable directly.
curl http://localhost:11434/api/versionIf Ollama is not running, start it.
ollama serveIf you set
params.baseUrl, make sure it includes Ollama’s/v1prefix.
Model not found
What’s happening:
The response returns a model not found error.
Why it’s happening:
The requested model has not been pulled into your local Ollama instance.
How to fix it:
List available models.
ollama listPull the missing model.
ollama pull llama3.2
Note
Local providers like Ollama usually run over HTTP and do not require llm.models[].auth or llm.models[].tls. If your Ollama endpoint is behind HTTPS or requires authentication, configure llm.models[].tls and llm.models[].auth like any other upstream provider.
