Streaming and async
Streaming moves the first token earlier; async lets one worker wait on several calls at once. Neither makes the model faster, and both change how a chain is written.
Latency is perceived
A response that takes four seconds to arrive feels slower than one that starts in half a second and finishes in five. Streaming does not reduce the total work; it moves the first visible token earlier, and the wait stops being empty. For a chat interface that is the difference between a product that feels alive and one that feels broken. A spinner communicates that something is happening; a stream communicates what is happening.
The metric to watch is time to first token, not only total latency. A stream that starts fast and stalls in the middle is worse than a steady one, because the user sees the stall. Measure the gap between chunks too, and treat a long gap as a failure even when the final answer is correct. Percentiles are more honest than averages here, because the slow tail is what people complain about. A slow first chunk is a product problem, not only an infrastructure one.
Three levels of stream
LangChain exposes three levels. stream and astream on a runnable yield the final output in pieces, which is what a chat UI needs. astream_events yields events for every step — model start, token, tool start, chain end — which is what a progress indicator or a trace needs. The names are close enough to confuse, so check which one the framework version actually exposes.
Pick the narrowest one that answers the question. Token chunks are cheap and simple; the event stream carries much more and costs more to process. A UI that only prints text does not need the event stream, and a tracer does not need to pretend it is a UI. The extra events are not free, and most of them are irrelevant to the screen. The event stream is also how a UI shows a tool call in progress before its result exists.
Tokens and events
async def main():
async for chunk in chain.astream({"topic": "streaming"}):
print(chunk, end="", flush=True)
async for event in chain.astream_events({"topic": "streaming"}, version="v2"):
if event["event"] == "on_chat_model_stream":
print(event["data"]["chunk"].content, end="", flush=True)Measure the wait for the first chunk
Run the consumer, then buffer the chunks and print only at the end. Compare the first-chunk time with the total — that gap is what streaming hides from the user.
Async is not a speed-up
Async does not make a model faster. It lets one worker wait on several calls at once, which matters because almost all of the time is spent waiting on the network. Three independent calls that each take a second can finish in about one second with asyncio.gather, and in three seconds one after another. The concurrency limit is the other half: gather everything at once and the provider rate-limits you.
The gain disappears the moment a blocking call enters the loop. A synchronous HTTP client or a time.sleep inside an async function stops every other task on that loop. Use the async client, or move the blocking work to a thread, and keep CPU-bound work out of the request path entirely. A thread pool is a bounded resource too, and an unbounded queue in front of it only hides the limit.
A synchronous call inside an async handler blocks every other request on the loop. The symptom is a service that is fast alone and slow under load.
Streaming with structure
A streamed JSON object is not valid JSON until the last brace arrives. Parsing each chunk fails, and concatenating the text and parsing at the end gives up the benefit. The usual answer is to stream the human-readable field and buffer the structured part, or to use a format whose records are line-delimited so each line can be parsed on arrival. Line-delimited JSON is the pragmatic choice when the reader needs structure before the stream ends.
Tool calls arrive in fragments too: a name in one chunk, arguments split across several. Accumulate by index and parse only when the model signals the call is complete. A partial argument object is not a smaller argument object; it is a different and usually invalid one, and acting on it is how a tool gets called with half a query. The safest rule is to act only on a complete object, even if that means the call is the last thing to appear.
Sequential and concurrent calls
Run both versions and compare the elapsed times. Then add a fourth call that raises and try `return_exceptions=True` to see how one failure is reported beside the successes.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
What does streaming actually improve?