Streaming

Streaming reduces perceived latency and lets clients process output incrementally. Set stream:true and consume the response as text/event-stream.

Chat Completions¶

curl -N https://api.koscompute.com/v1/chat/completions \
  -H "Authorization: Bearer $KOSCOMPUTE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.8-27b",
    "messages": [{"role": "user", "content": "Write a four-line poem about compilers."}],
    "stream": true,
    "stream_options": {"include_usage": true},
    "max_completion_tokens": 160
  }'

Each SSE frame begins with data:. Parse complete frames rather than network packets. The stream normally ends with data: [DONE]. Usage can appear in the final chunk when requested; intermediate chunks may have no usage.

Responses and Messages¶

Responses emits named lifecycle events; do not treat every delta as plain text. Anthropic Messages emits its own event names and typed content-block deltas. Use the SDK parser for the protocol you selected.

Production handling¶

• Disable proxy buffering in your own reverse proxy.
• Apply an idle timeout longer than expected pauses during reasoning.
• Accumulate tool argument fragments before parsing JSON.
• Treat a client disconnect as cancellation and avoid blindly replaying a non-idempotent workflow.
• If the connection ends without a normal terminal event, regard the result as incomplete.

Once response headers are sent, the server cannot replace HTTP 200 with a later 5xx. See streaming failures.