Choosing Open Embedding Models: A Reproducible Retrieval Evaluation

Choose an embedding model by testing retrieval on the questions and documents your application actually uses. A leaderboard result, vector dimension, or model size helps form a shortlist; none establishes the best model for every corpus.

This guide provides a comparison method and representative model families. It does not present a new benchmark ranking. Accuracy and latency numbers require a reproducible run with a fixed dataset, model revision, preprocessing, hardware, and result files.

Build a shortlist from model cards

Candidate	What to check before running
BGE	Retrieval instruction, language coverage, normalization, revision
E5	Distinct query and passage prefixes, truncation, normalization
Nomic	Task prefixes, context handling, revision-specific dimension support
MiniLM	Input truncation, domain fit, embedding dimension and license

Start with the official cards for BGE base English v1.5, E5 base v2, Nomic text v1.5, and all-MiniLM-L6-v2. These are concrete reference models, not a claim that they are the latest or strongest available options.

Open weights do not mean inference, hosting, or maintenance is free. Check the license and deployment constraints for each exact revision before adopting it. Avoid assuming that every model in a family has the same terms or input contract.

Get preprocessing right before comparing scores

For asymmetric retrieval, E5's documented recipe distinguishes queries from passages. Missing the required prefixes can invalidate a comparison. This helper makes that distinction visible in a test fixture:

def e5_input(text, kind):
    if not isinstance(text, str) or not text.strip():
        raise ValueError("Text is required")
    if kind not in ("query", "passage"):
        raise ValueError("kind must be query or passage")
    return kind + ": " + text.strip()

Do not apply that exact recipe to every model. Follow each card's instructions, use the same document content, and record truncation. If one encoder silently drops the relevant sentence, the comparison is partly testing input handling rather than semantic quality.

Fix the evaluation data

Create questions with labeled supporting document or chunk IDs. Include exact identifiers, paraphrases, domain terminology, ambiguous follow-ups, and questions with no supporting evidence. Keep a held-out set that is not used to tune thresholds or select prompts.

Record the corpus version and chunking strategy. A model tested on full documents cannot be compared fairly with another tested on carefully selected answer-sized chunks without explaining that difference.

Use a representative workload first. A public biomedical dataset may be useful for a specific experiment, but it cannot establish performance for your support tickets or code repository by itself.

Measure retrieval, then the answer

Candidate recall measures whether supporting evidence enters the pool. Rank-sensitive metrics measure where it appears. The reranking guide contains a small tested metric helper that makes these definitions explicit.

Measure answer support separately using the same answer model and prompt. A better retrieval score can fail to improve answers if context assembly truncates the evidence or the answer model ignores it. Likewise, an answer may look correct from prior model knowledge despite retrieving nothing useful.

Make latency and cost comparable

Report document-encoding throughput separately from query latency. Record batching, concurrency, device, precision, warm-up, cache state, and input lengths. For a remote service, record the region and include network time if that is what users experience.

Include index storage, updates, re-embedding, and operational work in the cost model. If a model change requires regenerating every stored vector, the migration cost belongs in the decision. A smaller vector may reduce raw storage without reducing encoder inference cost proportionally.

Publish the evidence with the conclusion

Keep model revision IDs, preprocessing code, question labels, ranked results, metric definitions, and run configuration with any reported ranking. Label a small pilot as a pilot and keep unavailable results missing rather than treating them as zero.

The preprocessing helper is locally tested. No embedding-model inference or new accuracy table is claimed in this revision. For dimension-specific tradeoffs, use the Matryoshka guide. For the full pipeline, start with the RAG tutorial.

If you would rather evaluate a managed retrieval service alongside the model shortlist, start a Supermemory pilot with the same corpus and questions. Compare supported answers and total operating cost separately from the embedding-only scores.