Every LLM system has two latencies. Most teams only watch one.
The two curves
- Time to first token (TTFT): how long until the user sees something. This is what "feel" is made of.
- Time to complete response (TTCR): how long until the whole answer is rendered. This is what "done" is made of.
A system can have great TTCR and terrible TTFT. It feels broken even when it isn't.
A system can have great TTFT and terrible TTCR. It feels fast but users leave before the answer finishes.
Where each goes wrong
Bad TTFT usually means:
- No streaming enabled on the client
- Long system prompt that isn't cached
- Heavy preprocessing before the LLM call
- Cold start on a serverless endpoint
Bad TTCR usually means:
- Output length isn't bounded
- Tool-use loop has runaway retries
- Backend is rate-limited and queuing
What to actually track
- p50 and p99 of TTFT, segmented by user-facing route.
- p50 and p99 of TTCR, segmented the same way.
- The ratio of tool-call latency to model latency, since tools are often the slow part and nobody notices.
One team I helped last quarter had great p99 TTCR of 3.2s, but p99 TTFT of 4.8s. Users were bouncing before the streaming even started. A single fix — moving the retrieved context through the cache — dropped TTFT p99 to 400ms. Retention went up 18%.
Watch both curves. Always.