Inference

Time to First Token (TTFT)

Time-to-first-token (TTFT) is how long before the first output token appears. TTFT heavily impacts perceived responsiveness in interactive applications. TTFT includes processing input, initializing KV cache, and generating the first token. Prefilling and other optimizations reduce TTFT. Batching increases TTFT since requests must wait for batch assembly. Interactive applications should optimize TTFT; batch applications care more about throughput. TTFT is often more critical than end-to-end latency for user experience.