SSE vs WebSockets for LLM Streaming APIs in FastAPI

LLM apps live or die on transport. Pick the wrong one and your chat UI sits behind a spinner while a reverse proxy quietly buffers the entire answer. Your reconnection logic then ships duplicate tokens to the screen, and the debugging session eats the afternoon. Two protocols cover nearly every case: Server-Sent Events and WebSockets. The choice decides proxy configuration, client code, and how much reconnection handling you write by hand.

TL;DR: Which One to Use

• SSE carries LLM output. OpenAI and Anthropic both stream responses over SSE. Their SDKs expose a token iterator and no WebSocket appears anywhere in either docs set.
• WebSockets earn their place when the client talks back mid-generation. Voice sessions, live agents that accept steering input, multi-user canvases: that is the short list.
• FastAPI ships SSE support. Import EventSourceResponse from fastapi.sse, pass it as response_class, yield events. Ten lines of glue.
• Streaming does not move your token bill. Same tokens, same price, different arrival time.
• Proxies buffer SSE until you disable buffering. nginx, Cloudflare, and several load balancers hold chunks back until the response finishes, which defeats the whole point of streaming.

What LLM APIs Send Over the Wire

Set "stream": true on an OpenAI Chat Completions request and the server answers with Content-Type: text/event-stream instead of a JSON blob. Each chunk of the reply arrives on its own data: line, and the connection ends with data: [DONE]. Anthropic keeps the same HTTP shape and gives events explicit names: message_start opens the stream, content_block_start and content_block_delta carry the growing text, message_delta reports stop reasons and cumulative usage, and message_stop closes the connection. ping events appear between them to keep the connection warm. The Anthropic streaming guide documents the full event flow, and the OpenAI Chat Completions reference covers the equivalent data: framing.

Token accounting stays identical between streaming and non-streaming calls. You pay for the same input and output tokens either way, and Anthropic reports the running total inside the final message_delta event. Streaming changes time to first token, not invoice size.

Connection limits follow ordinary HTTP rules. Each active stream holds one connection from client to server, tunneled through whatever proxy sits in the path. No upgrade handshake, no separate port, no special firewall exception. Local endpoints keep the pattern: Ollama serves newline-delimited JSON over its HTTP API, so a model on your laptop streams the same way it does in the cloud.

Also Read:

Ollama vs llama.cpp: Which Local LLM Runtime Should You Use?

Streaming Responses from FastAPI

FastAPI grew first-class SSE support in fastapi.sse. The response class sets text/event-stream as the content type, and anything you yield gets encoded onto the wire. Works with GET and POST alike, which is how MCP protocols stream events over POST requests too. This proxy endpoint pulls tokens from an upstream API and forwards them to the browser:

from fastapi import FastAPI
from fastapi.sse import EventSourceResponse, ServerSentEvent
import httpx, os

app = FastAPI()

@app.post("/chat/stream")
async def chat_stream(prompt: Prompt):
    headers = {"Authorization": "Bearer " + os.environ["OPENAI_API_KEY"]}
    body = {
        "model": "gpt-4.1-mini",
        "messages": [{"role": "user", "content": prompt.text}],
        "stream": True,
    }
    async with httpx.AsyncClient(timeout=120) as client:
        async with client.stream(
            "POST", "https://api.openai.com/v1/chat/completions",
            headers=headers, json=body,
        ) as upstream:
            async for line in upstream.aiter_lines():
                if not line.startswith("data: "):
                    continue
                payload = line[6:]
                if payload == "[DONE]":
                    yield ServerSentEvent(raw_data="[DONE]", event="done")
                    return
                yield ServerSentEvent(raw_data=payload, event="token")
                yield ServerSentEvent(comment="keep-alive")

Two details decide whether this works. First, raw_data instead of data: FastAPI JSON-serializes every data value, so passing an already-serialized JSON string lands double quotes on the wire and breaks your client parser. Second, the keep-alive comment. A comment line starts with a colon, EventSource clients ignore it, and the idle timer on your proxy resets. Long model generations pass through Cloudflare’s connection limits without a drop. Run it with uvicorn app:app --port 8000 after installing dependencies, and uv keeps that install step fast.

The browser side is a two-line consumer. const es = new EventSource("/chat/stream") plus es.addEventListener("token", e => render(JSON.parse(e.data))) handles the rest, including automatic reconnection when the connection drops. The FastAPI SSE tutorial covers ServerSentEvent fields, and the reference page lists every constructor argument.

Also Read:

MCP Server Security: How to Prevent Credential Leaks

SSE, WebSockets, and Polling Side by Side

The polling row deserves a note. Batching tokens client-side into five-second chunks cuts request count from fifty to one and lands within a tenth of a second of streaming latency. Polling loses on perceived responsiveness, not on raw cost.

Trust the transport only after you watch it under a real connection. Run curl -N against your endpoint with output piped to a line counter: the -N flag turns off curl’s own buffering, so what hits your terminal matches what the browser receives. Tokens arriving in separate bursts as they generate confirms the path is clean. One lump at the end means a proxy holds the response, and the fix belongs in proxy configuration rather than in application code.

When WebSockets Earn Their Keep

SSE plus ordinary POST requests already covers interrupt-and-steer patterns: the client hits /cancel while the SSE stream runs. The case for WebSockets starts when message frequency from many clients makes per-request overhead dominate, or when one connection must carry both directions of a shared session. Voice pipelines sit in that group, so do collaborative editors and game-state sync. Where WebSockets lose is operational: every load balancer needs the upgrade path allowed, every server process needs a connection registry, and every reconnect needs application-level resume logic you would get free from EventSource.

Also Read:

How to Write Structured AI Prompt Files for Complex Refactoring

The MCP ecosystem reinforces the SSE default. Protocol servers stream events over POST using the same framing, and a custom server built with FastAPI reuses EventSourceResponse without extra dependencies. If your integration target is an MCP server or a coding agent, SSE fluency matters more than WebSocket skill.

from fastapi import FastAPI, WebSocket

app = FastAPI()
sessions: dict[str, WebSocket] = {}

@app.websocket("/session/{sid}")
async def session(ws: WebSocket, sid: str):
    await ws.accept()
    sessions[sid] = ws
    try:
        while True:
            msg = await ws.receive_json()
            if msg.get("type") == "interrupt":
                sessions[sid] = ws  # cancel the in-flight generation
            await ws.send_json({"echo": msg})
    finally:
        sessions.pop(sid, None)

That skeleton carries the hidden bill: a process-local dictionary breaks the moment you run two workers, which pushes you toward Redis pub/sub for fan-out. SSE deployments never collect that complexity. When local models serve your traffic, transport choice interacts with where the model runs, and smaller models win on time to first token regardless of protocol.

Failure Modes That Show Up in Production

Proxy buffering. nginx holds the response until a buffer fills or the response ends. Streaming appears to work locally and then delivers in one burst through production. Set proxy_buffering off on that location and send X-Accel-Buffering: no from the app for safety. Cloudflare buffers small chunks too, which the keep-alive comment from the earlier example handles.

Also Read:

uv vs pip vs Poetry: Python Package Manager Comparison

Chunk boundaries versus event boundaries. HTTP chunks do not line up with SSE events. A JSON object can split across two reads, so a parser that decodes each chunk independently throws json.JSONDecodeError under load. Buffer by the blank-line frame separator, then decode. The MDN EventSource guide walks the framing rules.

Errors inside the stream. Anthropic emits error events, including overloaded_error, while the connection stays open. A handler that only scans for text deltas hangs the UI at exactly the moment the model falls over. Match on the event name and surface the message. Errors in the stream also come with metadata attached: the error event carries a type field that matches the standard errors API, so a router keyed on error.type picks the right user text without a second lookup. Code that treats a 200 response as proof of a healthy stream misses all of it, because these failures arrive as event payload rather than status code.

Replay duplication. EventSource reconnects on its own after a drop and replays from the last event ID. Servers that ignore Last-Event-ID reprint tokens the user already saw. Read the header, resume from the stored offset, or send unique IDs and let the client deduplicate.

Missing termination. A client waiting for [DONE] or message_stop spins forever when the server exits mid-generation without a closing event. Emit the sentinel in a finally block.

Final Thoughts

SSE is the transport for LLM output. OpenAI chose it, Anthropic chose it, FastAPI supports it natively, and every proxy problem it has has a documented fix. I reach for WebSockets only after a bidirectional requirement shows up in writing: voice, steering input at high frequency, or shared session state. I keep a copy of the event flow open while wiring a new provider, because the delta types differ enough between vendors to trip a parser. And I time first token before and after each config change, since that number exposes proxy buffering faster than any log line.

Start with one endpoint. Wire the FastAPI SSE route, drop an EventSource on the page, watch the first token land on screen, then tune proxies only if the burst pattern shows up. Transport debates are cheap in a prototype and expensive in production, and the prototype answer is almost always the production answer.