For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Configure TypeSafe Jev as an LLM provider for agentgateway.

Configure TypeSafe Jev, TypeSafe’s evaluation model, as an LLM provider in agentgateway. The Typesafe preset sends requests to TypeSafe in its native SystemOne format. Clients send OpenAI Decisions API requests, and agentgateway translates them to SystemOne before they reach TypeSafe.

The preset accepts only Decisions and SystemOne requests. Chat Completions, Responses, and Messages requests to a TypeSafe model fail.

Before you begin

1. Install and set up an agentgateway proxy.
2. Allow the AgentgatewayModel route kind on the http listener of the agentgateway-proxy Gateway. For the steps, see Enable LLM serving on a listener.

Set up access to TypeSafe

Store your TypeSafe API key in a Secret, and create a model that uses the Typesafe preset. The model attaches directly to the Gateway listener, so you do not create an AgentgatewayBackend or an HTTPRoute.

1. Get an API key from TypeSafe.

2. Save the API key in an environment variable.

   export TYPESAFE_API_KEY='<your-api-key>'

3. Create a Kubernetes Secret to store your API key.

   kubectl apply -f- <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: typesafe-secret
     namespace: agentgateway-system
   type: Opaque
   stringData:
     Authorization: $TYPESAFE_API_KEY
   EOF

4. Create an AgentgatewayModel that uses the Typesafe preset and references the Secret.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayModel
   metadata:
     name: typesafe
     namespace: agentgateway-system
   spec:
     parentRefs:
     - group: gateway.networking.k8s.io
       kind: Gateway
       name: agentgateway-proxy
       sectionName: http
     match:
       model: jev-latest
     provider: Typesafe
     policies:
       auth:
         secretRef:
           name: typesafe-secret
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Setting	Description
   spec.match.model	The model name that clients send in requests.
   spec.provider	The Typesafe preset. The preset supplies the default URL, https://api.typesafe.ai/v1, and the SystemOne request format.
   spec.policies.auth.secretRef	The Secret that holds your TypeSafe API key.

5. Send an OpenAI Decisions API request to the model. Agentgateway translates the request to the TypeSafe /v1/systemone API, and translates the response back to the Decisions format. The response answers each question in the request.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl "http://$INGRESS_GW_ADDRESS/v1/decisions" \
     -H 'Content-Type: application/json' \
     -d '{
       "model": "jev-latest",
       "input": [{"role": "user", "content": [{"type": "input_text", "text": "The package was left in the rain and the box is soaked through."}]}],
       "questions": [{"type": "predicate", "name": "visible_damage", "instructions": "Does the customer report visible damage to the package?"}]
     }' | jq

   curl "localhost:8080/v1/decisions" \
     -H 'Content-Type: application/json' \
     -d '{
       "model": "jev-latest",
       "input": [{"role": "user", "content": [{"type": "input_text", "text": "The package was left in the rain and the box is soaked through."}]}],
       "questions": [{"type": "predicate", "name": "visible_damage", "instructions": "Does the customer report visible damage to the package?"}]
     }' | jq

   Example output:

   {
     "model": "jev-latest",
     "answers": [
       {"type": "predicate", "name": "visible_damage", "probability": 0.64}
     ],
     "usage": {"input_tokens": 156, "output_tokens": 0, "total_tokens": 156}
   }

Next steps

For more information about Decisions requests, see Decisions. To declare the SystemOne format on a custom provider instead of the preset, see Custom providers.

Cleanup

kubectl delete AgentgatewayModel typesafe -n agentgateway-system
kubectl delete secret typesafe-secret -n agentgateway-system