Token Streaming Pipeline: LLM to UI at Scale

Vercel AI SDK handles SSE reconnection, state management (messages array), streaming text rendering, and error states out of the box. `useChat` reduces custom streaming client code from ~400 lines to ~10 lines. Works with any AI backend, not just Vercel.

Alternatives: Native EventSource API, TanStack Query + custom SSE hook, React Server Components (partial)

Vercel Edge Runtime has <50ms cold start (vs 200-800ms for Lambda) and supports Response with ReadableStream for native streaming. 130+ edge locations globally reduce latency by 40-80% vs a single-region server. Cost: $0.60/M edge function invocations. Critical: do NOT use Node.js runtime for streaming — it buffers responses.

Alternatives: Cloudflare Workers, AWS Lambda Response Streaming (Node 20), Fly.io long-running processes

SSE is simpler: browser-native, auto-reconnect, firewall-friendly (port 80/443), and works through HTTP proxies. WebSocket is required when you need bidirectional communication (client sends data mid-stream), or when streaming to mobile apps where EventSource is not natively available. 90% of LLM chat UIs should use SSE.

Alternatives: WebSocket via Socket.io, HTTP/2 server push (limited browser support), Long polling (fallback for proxies)

Claude Sonnet 4 streams at 60-100 tokens/second, fast enough for smooth UX. Gemini Flash 2.0 is fastest (150-200 tok/s) for latency-critical use cases. TTFT across providers: Claude 150-400ms, GPT-4o 200-500ms, Gemini 100-250ms. For fastest TTFT, use Groq (Llama 3.1 70B at 280 tok/s with 80ms TTFT).

Alternatives: OpenAI GPT-4o with stream=True, Google Gemini 2.0 Flash (150 tok/s), Together AI (open-source models, lower TTFT)

Web Streams API propagates back-pressure automatically through the chain: if the client isn't reading, the ReadableStream controller's desiredSize drops to 0 and the upstream reader pauses. This prevents the edge function from buffering the full LLM response in memory when a slow client can't keep up.

Alternatives: Node.js stream.pipeline(), Custom buffer with pause()/resume()

Serverless functions are stateless — at 10K concurrent streams, you need shared state for rate limiting and billing. Upstash Redis tracks connection counts per user ($0.20/100K commands). Cloudflare Durable Objects provide per-connection persistent state with geographic affinity for <20ms reads. Managed solutions (Ably) cost $0.0006/message but eliminate all infra work.

Alternatives: Ably (managed WebSocket gateway), Pusher Channels, Socket.io with Redis adapter on Fly.io

Mitigation: When the client closes the browser tab, the server continues consuming LLM tokens (and paying for them) until the response ends. Fix: listen for the request's `AbortSignal` in the route handler (`req.signal.addEventListener('abort', ...)`) and cancel the LLM API call immediately on client disconnect. This can save 20-40% of LLM costs for interactive use cases.

Mitigation: Middleware that buffers the full stream response (e.g., logging middleware, compression middleware) negates streaming and causes OOM on long responses. Audit every middleware in the request pipeline and ensure `passThrough` mode is used. Disable gzip compression for SSE routes — browsers handle it fine over plain-text SSE.

Mitigation: LLM provider returns a 429 after the stream has started. At this point, the HTTP status code has already been sent to the client (200 OK). Solution: send a specially formatted SSE error event (`event: error, data: {code: 'rate_limit', retryAfter: 60}`) and close the stream. The client's error handler should display a 'rate limit' message and offer a retry button.

Mitigation: Corporate firewalls, proxies (nginx), and some CDNs buffer SSE responses until they reach a certain size or the connection closes. Fix for nginx: set `proxy_buffering off`, `X-Accel-Buffering: no` header. For Cloudflare: the `Cache-Control: no-cache` header bypasses buffering. Always test streaming through your full network stack, not just localhost.