Streaming responses
Streaming response and settlement boundaries.
Only implemented streaming entries support stream
The currently supported streaming model-call entries are OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini streamGenerateContent.
Support matrix
| API family / operation | Streaming method | Notes |
|---|---|---|
| OpenAI Chat Completions | Request body "stream": true | Prefer usage from the stream; a conservative estimate may be used when the safety conditions are met. |
| OpenAI Responses | Request body "stream": true | Prefer response usage from the completion event. |
| Codex Search capability | Not supported | This is an independent transparent transport operation and does not promise a streaming response. |
| OpenAI Images Generations / Images Edits | Not supported | Image entries currently support synchronous responses only. |
| Anthropic Messages | Request body "stream": true | Observe usage from the Anthropic stream. |
| Gemini GenerateContent | POST /v1beta/models/{model}:streamGenerateContent | Use the native Gemini streaming path. |
| Anthropic Count Tokens | Not supported | Count Tokens is not a billable model-generation call. |
Gemini's special boundary
Gemini's non-streaming and streaming operations are different paths:
POST /v1beta/models/{model}:generateContent
POST /v1beta/models/{model}:streamGenerateContentPutting "stream": true in a generateContent request does not switch to the streaming path automatically; it returns stream_not_supported.
Settlement boundary
Failure before the response starts
For example, authentication failure, insufficient balance, a missing model, or no available upstream returns the corresponding error response to the client.
Failure during response delivery
A disconnect, client cancellation, or delivery failure may still settle trusted partial usage or output already observed by the gateway; settlement does not require the client to have successfully received a chunk. Without that billing evidence, no charge is made.
Settlement after the response ends
The server prefers valid usage. For token-priced streaming responses, when usage is missing or invalid, it uses a marked conservative lower-bound estimate only when the gateway can extract model output. Heartbeats, empty events, and raw transport chunks are not estimation evidence. Without trusted usage or extractable output, the reservation is released and the call is charged zero. If a 2xx response has already been submitted, its body may be empty or contain only SSE keepalives and cannot be rewritten as an error body.
Keep request_id in the client
A problem may occur after content has started streaming. Keep X-Request-Id while troubleshooting; it is more reliable than retaining only the last part of the output.
Last updated on