Advanced RAG: Route Questions Before Adding More Retrieval Stages
An advanced RAG pipeline should add work where a specific question needs it. Exact identifiers, vague descriptions, relationship questions, and follow-ups do not always benefit from the same retrieval path.
Begin with a baseline and classify its failures. Add a routing rule only when you can explain which query class it serves, what it costs, and how the application falls back when the classification is wrong.
Preserve exact information during query rewriting
A follow-up such as “Does that apply to the annual plan?” may need conversation context to become a useful retrieval query. Preserve the plan name, date, and account scope. A rewrite that sounds clearer but drops an identifier can retrieve the wrong source.
Keep the original question beside the rewrite in the trace. Evaluate both retrieval results so changes in source selection can be attributed. Do not allow rewriting to expand the user's permissions or turn untrusted source text into instructions.
Match the path to the missing evidence
| Failure | Candidate intervention |
|---|---|
| Exact product code is missed | Lexical lookup or structured filter |
| Relevant paraphrase is missed | Semantic retrieval |
| Correct source is buried | Reranking |
| Evidence needs an explicit join | Bounded relationship traversal |
| Retrieved passage lacks its conditions | Parent-section expansion |
These are hypotheses to test, not mandatory stages. Adding every component increases cost and creates more places for evidence to disappear.
Set a budget for each branch
Bound candidate count, traversal depth, model calls, and elapsed time. Decide whether a slow branch returns a partial supported answer, asks a clarifying question, or falls back to a simpler path.
Record the evidence supplied by each branch and deduplicate by source identity and version. Two similar passages from the same old document should not outweigh one current authoritative record simply because they arrive through different retrievers.
Compare incremental value
Run the same labeled questions with the baseline, then with one intervention. Measure supporting-evidence coverage, final answer support, latency, and tokens by query class. A better overall average can hide regressions in exact-ID questions.
Before serving users, run the RAG chatbot launch review across evidence quality, access, and degraded behavior.
The hybrid-search guide and reranking evaluation cover two of these branches. For a managed retrieval comparison, try Supermemory's documented search modes with your fixed question set. Start a small pilot and keep only the complexity that improves a demonstrated failure.