SilkRouter Streaming API and Latency Guide

When streaming helps

Streaming does not always reduce total model time, but it can reduce perceived latency. Users see the answer begin sooner, which matters for chat, long summaries, code generation, and agent progress updates.

Use streaming when partial output is useful. For strict JSON, short classifications, or hidden background jobs, a normal response may be simpler and safer.

Backend handling

Treat streaming as a stateful request. Your backend should forward chunks, detect completion, handle disconnects, and record the final status even if the user closes the tab.

Keep retry behavior conservative. Retrying after partial output can create duplicate text or extra cost unless the UI and request ID are designed for it.

• Track first-token latency and total completion time separately.
• Capture final token usage when the stream completes.
• Set server timeouts higher than expected model duration.
• Handle client disconnects without leaking running requests.

UI states

A good streaming UI needs clear states: connecting, generating, completed, interrupted, and failed. Users should know whether an answer is still coming or whether they need to retry.

Avoid hiding errors after partial output. If the stream fails, show that the answer may be incomplete and offer a controlled retry path.

Latency monitoring

Monitor latency by workflow and model, not only as one global average. Streaming performance can vary by prompt size, output length, provider health, and fallback rate.

SilkRouter usage views help teams connect model choice and request behavior to operational outcomes instead of debugging only from browser timing.