Rate limits and capacity

Not every 429 means the same thing. Read the error code, message, metadata, and Retry-After header.

Per-key request rate¶

A key can have a requests-per-minute policy. Exceeding it returns 429 and normally a short retry interval. Reduce request rate, use a client-side token bucket, or request a different policy.

Per-key concurrency¶

A key can limit in-flight requests. The error metadata can include concurrent_limit and error_code:"concurrent_limit_exceeded". Wait for an existing request to finish before retrying.

Model and global capacity¶

When all usable slots are busy, the router returns 429 with sanitized capacity information. A queue can admit work briefly; expiration in that queue is also 429, not a generation timeout. Do not infer internal topology from the capacity object.

Modality capacity¶

Vision can have a separate capacity pool from text. Audio and speech workers can likewise saturate independently. GET /v1/models exposes current public capacity and modality splits where available. These live values are informational and can change immediately; they are not reservations or guarantees.

Temporary readiness changes and free-capacity changes are normal operational state, not documentation drift.