Retries

Retry only failures likely to be temporary: 429, 502, 503, and 504. Do not retry an unchanged 400, 401, 403, 404, or 413 request.

Recommended policy¶

1. Use Retry-After when present.
2. Otherwise start near one second and increase exponentially.
3. Add random jitter so clients do not synchronize.
4. Set a total deadline and a small attempt limit.
5. Cancel work the user no longer needs.

import random
import time

delay = retry_after_seconds or min(8.0, 0.8 * (2 ** attempt))
time.sleep(delay * random.uniform(0.8, 1.2))

Replay safety¶

Text generation is not idempotent: a retry can produce a different answer. A connection can also fail after the model generated output. Before replaying tool-driven or side-effecting workflows, determine whether your application already executed any returned tool call.

KosCompute does not currently expose an idempotency-key contract for generation. Enforce idempotency in your application around external side effects.