For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
LLM observability
Prompt logging, cost tracking, and audit trail via Langfuse, LangSmith, and more
Agentgateway can send LLM telemetry to specialized observability platforms for prompt logging (request/response logging), cost tracking, audit trail, and performance monitoring.
How it works
Agentgateway exports LLM telemetry via OpenTelemetry, which can be forwarded to LLM-specific observability platforms. These platforms provide the following.
- Prompt/response logging - Full request and response capture (also known as request logging, audit trail).
- Token usage tracking - Monitor costs across models and users (also known as cost tracking, spend monitoring).
- Latency analytics - Track response times and identify bottlenecks.
- Evaluation - Score and evaluate LLM outputs.
- Prompt management - Version and manage prompts.
Configuration
Set up OpenTelemetry tracing to export LLM-specific telemetry. See the OpenTelemetry stack setup guide for details.
Agentgateway automatically includes these LLM-specific trace attributes.
| Attribute | Description |
|---|---|
gen_ai.operation.name | Operation type (chat, completion, embedding). |
gen_ai.request.model | Requested model name. |
gen_ai.response.model | Actual model used. |
gen_ai.usage.input_tokens | Input token count. |
gen_ai.usage.output_tokens | Output token count. |
gen_ai.provider.name | LLM provider (openai, anthropic, etc.). |
Was this page helpful?
