Every model release that bumps the context window triggers the same headline. RAG is dead. Long context wins. And then a week later teams quietly go back to retrieving.
The real economics
Long context is not free. Putting a million tokens in front of the model costs real money per call — and crucially, costs per call, every time. A cache hit on a retrieved chunk is measured in micro-cents; re-feeding the same giant corpus is measured in dollars.
That's before we talk about latency. First-token time scales with context length. A chatbot that takes four seconds to start responding loses to one that takes 300ms, even if the answer quality is marginally worse.
Where long context actually wins
Long context shines when:
- The relevant passage location is genuinely unknown in advance
- The task requires cross-document reasoning you can't shard
- You only run the query once — one-off analysis, not interactive chat
Retrieval shines everywhere else. And "everywhere else" is still ~90% of production deployments.
The synthesis
The teams getting this right aren't picking one. They're using retrieval to shrink the working set to a few thousand tokens, then letting long context carry the reasoning over those. Hybrid, not either-or.