For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Route OpenAI Decisions API requests through agentgateway and track token usage.

Route OpenAI Decisions API requests through agentgateway to get token usage tracking and observability.

About

The OpenAI Decisions API answers a list of questions about an input. Each question is a predicate, a choice, or a score, and the response returns one answer for each question.

The following providers accept Decisions requests. Other providers reject them.

• OpenAI accepts Decisions requests natively.
• TypeSafe accepts them after agentgateway translates them to the SystemOne format.
• Azure accepts them for Azure AI Foundry resources, after agentgateway translates them to the SystemOne format.
• Custom providers accept them when they declare the Decisions or SystemOne format.

Agentgateway processes a request as a Decisions request only when the request path maps to the Decisions route type. Review the following behavior of Decisions requests.

Before you begin

1. Install and set up an agentgateway proxy.
2. Set up access to the OpenAI LLM provider.

Map the Decisions route type

Requests default to the Completions route type, so map the /v1/decisions path to the Decisions route type on your OpenAI AgentgatewayBackend. If you already set up multiple endpoints, add the /v1/decisions path to your existing AgentgatewayBackend instead.

1. Update the openai AgentgatewayBackend from the OpenAI setup guide. This command replaces that AgentgatewayBackend.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayBackend
   metadata:
     name: openai
     namespace: agentgateway-system
   spec:
     ai:
       provider:
         openai: {}
     policies:
       auth:
         secretRef:
           name: openai-secret
       ai:
         routes:
           "/v1/decisions": "Decisions"
           "/v1/chat/completions": "Completions"
           "*": "Passthrough"
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Setting	Description
   ai.provider.openai	The OpenAI provider. The example sets no default model, so each request must name its model.
   policies.ai.routes."/v1/decisions"	Processes requests on this path as Decisions requests. Without this entry, the "*": "Passthrough" route forwards Decisions requests without processing them. If you omit routes, agentgateway processes every request as a chat completion.
   policies.ai.routes."*"	Passes requests on all other paths through unchanged.

Send a Decisions request

Send a request with an input and a list of questions to the /v1/decisions path.

1. Send the request. Replace <decisions-model> with an OpenAI model that supports the Decisions API.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl "http://$INGRESS_GW_ADDRESS/v1/decisions" \
     -H 'Content-Type: application/json' \
     -d '{
       "model": "<decisions-model>",
       "input": [{"role": "user", "content": [{"type": "input_text", "text": "The package was left in the rain and the box is soaked through."}]}],
       "questions": [{"type": "predicate", "name": "visible_damage", "instructions": "Does the customer report visible damage to the package?"}]
     }' | jq

   curl "localhost:8080/v1/decisions" \
     -H 'Content-Type: application/json' \
     -d '{
       "model": "<decisions-model>",
       "input": [{"role": "user", "content": [{"type": "input_text", "text": "The package was left in the rain and the box is soaked through."}]}],
       "questions": [{"type": "predicate", "name": "visible_damage", "instructions": "Does the customer report visible damage to the package?"}]
     }' | jq

   Example output:

   {
     "model": "<decisions-model>",
     "answers": [
       {"type": "predicate", "name": "visible_damage", "probability": 0.64}
     ],
     "usage": {"input_tokens": 156, "output_tokens": 0, "total_tokens": 156}
   }

Verify token tracking

After the request completes, check that agentgateway recorded the token usage in its metrics.

1. Port-forward the agentgateway proxy on port 15020.

   kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 15020 &

2. Open the agentgateway metrics endpoint.

3. Look for the agentgateway_gen_ai_client_token_usage metric. The metric includes labels for the token type (input or output) and the model.

For more information about LLM metrics, see LLM cost tracking.

Cleanup

To restore the original AgentgatewayBackend, apply the openai AgentgatewayBackend from OpenAI again. To keep the AgentgatewayBackend but stop processing Decisions requests, remove the "/v1/decisions": "Decisions" entry from its routes.