Streaming chat completions: handling server-sent events that survive production

What the wire format looks like

With stream: true the response has Content-Type: text/event-stream and the body is a series of events separated by blank lines. Each event is a data: line holding a JSON chunk shaped like a normal completion, except that choices[0] has a delta object instead of a message:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Token"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The first chunk usually carries only the role, the middle chunks carry text, the last carries finish_reason, and [DONE] closes the stream. Tool calls stream the same way, with delta.tool_calls[].function.arguments arriving as string fragments you concatenate before parsing. The format is identical to the OpenAI API, so the same streaming code works against every model id on FastInfra.

Consuming the stream with the SDKs

The official SDKs hide the parsing:

stream = client.chat.completions.create(
    model="qwen/qwen3.6-27b",
    messages=messages,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        sys.stdout.write(chunk.choices[0].delta.content)
    if chunk.usage:
        log_usage(chunk.usage)

In the browser, fetch plus a ReadableStream reader is enough; split on blank lines, strip the data: prefix, and JSON.parse each payload. Avoid EventSource: it only supports GET and cannot send a JSON body or an Authorization header.

Time to first token versus total latency

Streaming does not make generation faster; it moves the moment the user sees progress. Two numbers matter. Time to first token (TTFT) is prompt processing plus queueing, and grows with input length. Tokens per second is the generation rate, and depends on the model size and the hardware behind the route. For a chat UI, a TTFT under one second and 30+ tokens per second feels instant; a 70B model on a busy route may deliver a lower rate, so shorter prompts and smaller models remain the main levers. FastInfra reports usage for streamed requests when you set stream_options.include_usage, so cost tracking does not depend on counting chunks.

Timeouts: per chunk, not per request

A single total timeout fails long answers that are progressing normally. Use two:

Set max_tokens so the ceiling is rarely reached; it is cheaper to bound output in the request than to abort it on the client. Proxies in front of your service (and in front of the API) often have their own idle limits near 100 seconds, which is another reason to keep chunks flowing rather than waiting on a single long response.

Detecting incomplete streams

The protocol gives you two explicit end signals: a chunk with a non-null finish_reason and the [DONE] sentinel. If the connection closes before either, you received a partial answer. Do not render it as complete and do not store it as a successful completion; surface a retry affordance. A finish_reason of length means the model hit max_tokens, which is a complete response that was truncated by your own limit, not a transport failure.

Backpressure and cancellation

Browsers and server frameworks will buffer chunks you are slow to consume; for most chat UIs that is fine, but if you forward the stream to a client, propagate the client's disconnect upstream. In the SDKs, closing the iterator (or aborting the AbortController in JavaScript) closes the HTTP connection and stops generation on the server. Abandoned streams that keep running still generate output tokens that you pay for, so cancellation is a cost control as well as a resource one. Rate limits still apply per request; a cancelled stream counts as one request against the published limits.

Retries that do not duplicate output

Retry before the first content chunk freely: nothing has been shown and nothing has been generated. After content has arrived, a blind retry restarts generation and produces different text; the user would see the answer rewrite itself. The usual compromise is to retry only on failures before first token, and to offer a manual "regenerate" action for failures mid-stream. With 785 model ids behind one endpoint, a second strategy is a fallback to a smaller, faster id for the retry so the user is not kept waiting twice.

Testing streaming before it reaches users

Streaming bugs rarely appear in a happy-path demo; they appear under slow networks, long answers and mid-stream failures. Build a small harness that replays recorded event streams through your parser with three deliberate faults: a chunk split across two TCP reads (your line buffer must not assume one event per read), a stream that ends without [DONE], and a chunk whose finish_reason is length. Each should produce a distinct, observable outcome: correct reassembly, an "incomplete" state, and a "truncated by max_tokens" state.

Then test against the real endpoint with one of the 12 free-tier model ids in the model catalog, which costs nothing and exercises the identical wire format as paid models. Measure time to first token and tokens per second for each candidate id on the pricing page; the numbers belong next to price in your model-selection spreadsheet, because a cheaper model that streams slowly can feel worse than a dearer one that answers in half a second. Repeat the measurement at your peak hour, since route load changes tokens per second more than model choice does, and keep the harness in CI so a parser regression is caught before a user sees a half-rendered answer.