For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Configure tracing in agentgateway: enable OTLP export, set up authentication and TLS, control the sampling rate, filter spans, and customize span attributes.

Agentgateway natively exports distributed traces over OTLP (OpenTelemetry Protocol). Traces include HTTP, MCP, and LLM spans with attributes that follow the OpenTelemetry semantic conventions for generative AI.

Enable tracing

Agentgateway has two places that you can configure tracing.

To enable tracing in agentgateway, add a tracing block under the frontendPolicies section and point agentgateway to your OTLP-compatible backend. For sample tracing backend setups, see Sample tracing configurations.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true

Set up authentication and TLS

When connecting to an OTLP backend that requires authentication, such as a SaaS observability platform, use the policies.requestHeaderModifier section to add required authentication headers and policies.backendTLS to enable TLS for the connection.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: <your-otlp-endpoint>.com:443
    protocol: http
    randomSampling: true
    policies:
      backendTLS: {}
      requestHeaderModifier:
        set:
          Authorization: "Basic <api-key>"

Control sampling rate

Use the randomSampling setting to control the fraction of requests for which spans are exported. Set randomSampling: true to sample 100% of requests, or provide a decimal between 0 and 1 for a percentage.

Two sampling settings decide whether a request is traced, and which one applies depends on the incoming request.

Because clientSampling defaults to true, agentgateway continues a trace that a client already started even when randomSampling is false. Set clientSampling to false or to a decimal to sample those requests instead.

In the following example, you want to sample 10% of requests.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: 0.1   # sample 10% of requests

Filter spans

Use the filter field to write a CEL expression that controls which sampled spans are exported. A span is exported only when the expression evaluates to true. The filter runs after sampling, so it only evaluates spans that were already selected by randomSampling.

The following example exports only spans for requests with an HTTP response code of 400 or greater.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
    filter: 'response.code >= 400'

You can combine conditions to export specific traffic. The following examples show common filter patterns:

For the full list of available CEL variables, see the CEL variables reference.

Customize span attributes

Agentgateway emits standard OpenTelemetry attributes as shown in Default span attributes. You can add custom attributes to your spans or remove default ones. Note that customizing span attributes does not work on policy call child spans. For more information, see Policy call child spans.

Add span and resource attributes

Use the attributes field to add custom tags to individual spans. Attribute values are evaluated as CEL expressions on every request, so they can be dynamic. For example, you can use a CEL expression to add the user ID from a request header or the HTTP response code.

Use the resources field to describe the agentgateway process itself. Resource values are static values that are added to every span. For example, you can add the name of the agentgateway process, the environment it runs in, what version it is. Your tracing backend uses resource attributes to label and group spans in its service list. The most common resource attribute is service.name, which sets the name that is displayed in your tracing backend. It defaults to agentgateway if not set.

The following example sets a custom service name and deployment environment in resources, and adds the user ID and request path to each individual span by using attributes.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
    resources:
      service.name: '"my-agentgateway"'
      deployment.environment: '"production"'
    attributes:
      user.id: 'request.headers["x-user-id"]'
      request.path: 'request.path'

Remove span attributes

Use the remove field to drop attributes from spans before your attributes expressions are applied. This setting is useful for stripping default attributes that are redundant or that you do not want to export.

The following example removes the HTTP version and source address from the span.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
    remove:
      - src.addr
      - http.version

Set process-level tracing defaults

The config.tracing section sets tracing defaults for the agentgateway process itself. Agentgateway reads the config section only at startup, so a change to this section requires a restart. Note that the field names differ from frontendPolicies.tracing: the endpoint is otlpEndpoint rather than host, and the protocol is otlpProtocol rather than protocol.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  tracing:
    otlpEndpoint: http://localhost:4317
    otlpProtocol: grpc
    randomSampling: true
    clientSampling: true

A randomSampling or clientSampling value that you set in frontendPolicies.tracing overrides the value in config.tracing for the requests that the frontend policy handles.