For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Fault injection

Inject artificial latency into requests to test how your clients and services handle slow responses.

Inject artificial latency into requests with an AgentgatewayPolicy to test how your clients and upstream services handle slow responses. Fault injection is useful for verifying that timeouts, retries, and client-side deadlines behave as expected. To learn more, you can use fault injection alongside Timeouts and Retries.

The injected delay counts against the request timeout. If the delay is longer than the configured request timeout, the request times out.

Before you begin

1. Set up an agentgateway proxy.
2. Install the httpbin sample app.

Inject a delay

1. Create an HTTPRoute for the httpbin app.

   kubectl apply -n httpbin -f- <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: httpbin-delay
     namespace: httpbin
   spec:
     hostnames:
     - faultinjection.example
     parentRefs:
     - name: agentgateway-proxy
       namespace: agentgateway-system
     rules:
     - backendRefs:
       - kind: Service
         name: httpbin
         port: 8000
   EOF

2. Create an AgentgatewayPolicy that injects a fixed 2-second delay into the requests that the HTTPRoute handles.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayPolicy
   metadata:
     name: httpbin-delay
     namespace: httpbin
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: httpbin-delay
     traffic:
       delay:
         duration: 2s
   EOF
   Review the following table to understand this configuration.

   Field	Description
   targetRefs	The route, gateway, or listener to apply the delay to. In this example, the policy targets the httpbin-delay HTTPRoute.
   traffic.delay.duration	The latency to inject before the request is forwarded to the backend. Set either a duration string, such as 2s or 500ms, or a CEL expression that is evaluated against the request. For more information, see Inject a probabilistic or random delay.

3. Send a request along the route and verify that the response is delayed by about 2 seconds.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl -s -o /dev/null -w "%{time_total}s\n" http://$INGRESS_GW_ADDRESS/get -H "host: faultinjection.example"

   curl -s -o /dev/null -w "%{time_total}s\n" localhost:8080/get -H "host: faultinjection.example"

   Example output:

   2.01s
   

4. Optional: Verify that the delay policy is applied in the proxy configuration.

   1. Port-forward the gateway proxy on port 15000.

      kubectl port-forward deploy/agentgateway-proxy -n agentgateway-system 15000

   2. Find the delay policy in the config dump.

      curl -s http://localhost:15000/config_dump | jq '[.policies[] | select(.policy.traffic.delay?)] | .[0]'

Inject a probabilistic or random delay

Because duration accepts a CEL expression, you can inject latency into only a subset of requests, or add jitter. The expression is evaluated against each request and returns either a duration or a number that is interpreted as milliseconds. A non-positive result injects no delay.

For example, the following policy delays approximately 10% of requests by 500ms.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: httpbin-delay
  namespace: httpbin
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: httpbin-delay
  traffic:
    delay:
      duration: "random() < 0.1 ? 500 : 0"
EOF

Cleanup

You can remove the resources that you created in this guide. Run the following commands.

kubectl delete httproute httpbin-delay -n httpbin
kubectl delete AgentgatewayPolicy httpbin-delay -n httpbin