Contextual Reranking for RAG: Query Context, Candidates, and Tests

Contextual reranking means scoring retrieved candidates against a clearly defined question that includes the relevant conversational context. It is distinct from enriching chunks before indexing or combining lexical and vector retrieval. Keep those stages separate so you can identify which change helped.

A reranker can reorder candidates; it cannot recover a document that never entered its candidate set. Start by checking whether the evidence exists in the retrieved pool before changing the ranking model.

Resolve the question before scoring candidates

If the user asks “does that also apply in Germany?”, the literal query lacks the subject of “that.” Construct a standalone retrieval question from the relevant current conversation, preserving constraints such as product, jurisdiction, version, and time. Do not indiscriminately attach every stored user preference.

Record both the original question and the resolved query in the evaluation trace. A bad rewrite can confidently retrieve evidence for the wrong subject. Include tests where the previous subject changes or where the user rejects an earlier assumption.

Distinguish four operations

Operation	Changes	Evaluation question
Query resolution	The search question	Was the user's meaning preserved?
Chunk enrichment	Indexed document text	Does added context identify the source accurately?
Candidate retrieval	The eligible evidence pool	Is supporting evidence present?
Reranking	Order within that pool	Is useful evidence near the top?

Anthropic's contextual retrieval research describes an evaluated combination of techniques. Its results are attached to that setup; they are not a guaranteed improvement for every conversational reranker or Supermemory application.

Enforce scope before ranking

Filter candidates to the authorized user, workspace, and document permissions before sending their text to a ranking service. A final UI filter is too late if unauthorized content already reached the model or another provider.

Use stable document and chunk IDs so evaluation can compare the same evidence before and after ranking. If content changes, record the version. Otherwise, a score change may reflect a changed corpus rather than a better ranking method.

Use a small metric implementation you can inspect

This helper computes reciprocal rank and evidence recall at a cutoff. It removes duplicate candidate IDs, rejects unanswerable cases from this particular calculation, and makes no assumptions about the ranking model.

def retrieval_metrics(ranked_ids, relevant_ids, k=5):
    if not isinstance(k, int) or isinstance(k, bool) or k <= 0:
        raise ValueError("k must be a positive integer")
    relevant = set(relevant_ids)
    if not relevant:
        raise ValueError("Score unanswerable questions separately")
    unique = list(dict.fromkeys(ranked_ids))[:k]
    hits = [i + 1 for i, item in enumerate(unique) if item in relevant]
    return {
        "recall_at_k": len(set(unique) & relevant) / len(relevant),
        "reciprocal_rank_at_k": 1 / hits[0] if hits else 0.0,
    }

For relevant IDs A and B, ranking X, A, A, B gives recall 1 and reciprocal rank 0.5 at a cutoff of three unique candidates. These are deterministic metric calculations, not measured model performance.

Compare one change at a time

Keep the same corpus version, labels, candidate budget, and answer model. Compare the baseline, query resolution alone, reranking alone, and the combination. If testing chunk enrichment too, add it as a separately recorded condition.

Track candidate recall before reranking, top-result quality afterward, final answer support, latency, and context tokens. Include unanswerable questions with a separate measure of whether the system correctly declines to infer an unsupported answer. Do not average those cases into a metric that requires at least one relevant document.

Decide from the failures

If the right document is absent, improve ingestion, filters, chunking, or candidate generation. If it is present but buried, reranking may help. If it reaches the model but the answer is wrong, inspect the generation prompt and source interpretation.

The metric helper is tested locally on duplicate IDs, no hits, multiple relevant documents, and invalid cutoffs. No new reranker benchmark was run. Combine this evaluation method with the hybrid-search guide and RAG tutorial when building the actual pipeline.

To evaluate a managed retrieval path, review Supermemory’s search API and run your labeled questions through its documented options. Score the returned evidence against the same baseline; use the results to decide whether further ranking work is needed.