How to Evaluate Memory Services for a Multi-Tenant AI Product

Evaluate a memory service for multi-tenancy with adversarial access cases and mixed workloads, not just a successful lookup for one user. Logical isolation and predictable performance are separate requirements, and both should appear in the evaluation.

This is a selection framework, not a ranking of providers. The best fit depends on your identity model, deployment constraints, source systems, and operating requirements.

Write the access matrix first

List personal, project, and organization scopes, then specify who can read, write, correct, and remove records in each. Include guests, users who leave a project, and background service accounts.

Turn each denied action into an acceptance case. A fluent answer that omits an explicit source citation can still expose another tenant's information, so inspect retrieved evidence and generated output separately.

Test the same identifiers under different tenants

Two customers may both have a project called “Launch” and a document called “Roadmap.” Test that their records remain distinct through ingestion, retrieval, revision, and deletion. Stable source names are not globally unique identities.

Also test cache keys and asynchronous jobs. A service can enforce scoped retrieval while your application accidentally caches the result under an unscoped query string.

Measure workload interference

Run a large ingestion job for one tenant while another issues normal queries. Record latency, errors, queue delay, and readiness by tenant. An overall average can conceal a poor experience for the smaller customer.

Ask which quotas, scheduling controls, and deployment options are available in the exact offering under review. Treat service commitments as evidence to verify, not assumptions inferred from the word “enterprise.”

Include exit and incident operations

Check whether you can export one tenant's data, identify its derived records, revoke access, and remove it without affecting another tenant. Test an interrupted removal workflow and how incomplete work is reported.

For a Zep candidate, use the architecture evaluation guide to separate documented service behavior from application responsibilities.

The container authorization guide explains why a scope identifier still needs trusted application enforcement.

Use the isolation implementation guide to build the fixture and the memory API comparison for broader selection criteria. To evaluate Supermemory, start a two-tenant pilot and keep the same gates for every candidate.