A RAG demo is easy. A RAG system that stays accurate after six months of growing data, drifting user intent, and changing source documents is hard. The difference is almost entirely in the boring details.
Chunking is product work, not infra
How you split your documents determines what your model can answer. Chunk by semantic boundary, not by token count. Keep parent context available so retrieved chunks aren't orphans.
Hybrid retrieval beats pure embeddings
BM25 + vector with a reranker on top consistently beats either alone. Don't be precious about going back to lexical search for keywords and IDs.
Freshness, caching, and the rest
Have a re-index pipeline. Cache aggressively for repeat questions. Log every retrieval so you can debug why an answer went wrong in production.