Company memory, measured
We benchmarked Open Passport against Mem0, Graphiti (Zep), Supermemory and vector search on what a company needs from memory. On the one held-out test, it lost. This is everything we measured.
Open Passport is a company memory: one markdown vault that every agent, department and app reads from. Facts are plain lines with dates and sources. Departments keep their own notes private. Old values move to a dated history instead of disappearing.
Memory products publish benchmarks, but almost all of them measure one person talking to one assistant. We wanted to know whether a memory built for a company actually does better at a company's work. So we built a benchmark for that, pre-registered it, tuned on a development set, and then ran a sealed test once.
IIWhat makes company memory different
A personal-chat benchmark asks whether the assistant remembers what you said. A company asks harder questions:
- Many people and departments. Sales, support and engineering each know different things.
- Decisions and provenance. What was decided, by whom, and where it came from.
- Facts that change. Prices, owners and policies move; the right answer depends on the date.
- House style. Answers should sound like the company, not like a chatbot.
- Privacy between departments. One team's confidential facts must not reach another.
- Forgetting on request. Erase a fact, and nothing derived from it is left to resurface.
- Tokens per answer. Memory is loaded on every call, so its size is a running cost.
IIIThe held-out test
The test ran once, on 2026-10-26, on data sealed before Open Passport was frozen. The pre-registered rule asked Open Passport to beat Mem0 and per-department vector search overall. It did not, under both judges and both bootstrap seeds.
The decisive loss was real organisational history: questions about the Kubernetes enhancement-proposal record, which no system had seen before the test.
It won elsewhere on the same test. These are the pre-registered rows, as points of Open Passport minus the comparator, with 95% confidence intervals:
| Test | vs Mem0 | vs per-dept vector |
|---|---|---|
| Real company history n 283 | −20.8 [−27.2, −14.5] | −26.5 [−32.9, −20.1] |
| Durable facts after 24 months n 300 | −4.3 [−7.0, −1.7] | +2.7 [−1.0, 6.3] |
| Rare old facts after 24 months n 300 | −3.0 [−6.0, 0.0] | +34.7 [28.0, 41.0] |
| Stale answers, lower is better n 300 | −1.7 [−5.0, 1.7] | −15.3 [−20.3, −10.3] |
| Company Q&A from raw history n 600 | +0.7 [−2.5, 4.0] | +16.2 [12.0, 20.3] |
| Current preferences, asker known n 576 | +12.2 [6.9, 17.5] | +18.8 [13.7, 24.0] |
| House style and taste n 180 | +37.6 [33.6, 41.5] | +18.0 [14.2, 21.9] |
| Erasure, incl. derived summaries n 58 | 55/58 vs 1/58 | 55/58 vs 46/58 |
| Tokens per answer, ratio n 3,103 | 1.043× | 0.403× |
Underlined numbers are significant losses; white ones are significant wins after Holm correction. Mem0 used slightly fewer tokens than Open Passport; per-department vector search used about two and a half times as many.
Open Passport is better at the company-shaped things: what someone prefers now, how the company writes, and forgetting completely. It is worse at the thing that matters most, answering from a large, messy history it has never seen. That decided the test.
IVWhere it leads in development
Everything in this section is development-set data from studies 2, 3 and 5: measured on data Open Passport was tuned on. It is indicative, and it favours Open Passport.
Private, and still useful
With every system's own per-department scoping switched on, the others leaked 0–1 of 24 probes and Open Passport leaked 0, which is level. The difference is that scoping left them unable to answer any of the 16 questions that need a fact another team allows to be shared. Open Passport answered 14.
House style
Answers in the company's own voice: 89.5 for Open Passport against 46.8 for Mem0, 47.6 for Graphiti, 79.1 for Supermemory and 52.0 for plain vector search. A plain written style guide pasted into the prompt scored 91.5, level with us, so the honest claim is narrower: Open Passport learns the style without anyone writing the guide.
Dates and changes
On curated notes, the current value after one to four changes: 32/32, against 29 for Mem0 and 8 for Graphiti. "As of date X": 18/20, against 11 for Mem0 and 1 for Graphiti. Plain vector search matched or beat us on the date questions: 19/20 on "as of date X", and 20/20 against our 19/20 on "what changed since X".
Junk writes
After raw ingest, Open Passport kept 4 of 39 junk and poison writes. Mem0 kept 24, Graphiti 6.
VWhere it loses
- Real history. Held-out, 143/300 against 205 for Mem0. On Python enhancement-proposal history in development, per-department vector search was ahead too: 97.3 against 91.8.
- Durable facts after two years. Held-out, 4.3 points behind Mem0.
- Rare facts. A fact mentioned once, asked about two years later: 82 for Open Passport, 88 for Mem0 (development).
- Stale answers. Answers that used a value that had since changed: 6.7% for Open Passport, 1.7% for Mem0 (development).
- Preferences when nobody says who is asking. Held-out, per-department vector search 27.1, Open Passport 19.8, Mem0 12.3. And putting the whole chat history in the prompt beat Open Passport on current preferences by 11.1 points.
- Fresh conflicting writes. When an app wrote a newer, correct value, Open Passport held it for review and served 0 of 12; Mem0 served 7.
- Forgetting. Importance-based forgetting archived nothing in 24 months of activity.
- LoCoMo. On long personal chats, which is not what Open Passport is for, it is about 22 points behind Mem0.
VIMethod
- Studies were pre-registered. The held-out test (study 4) ran once, on data sealed before Open Passport was frozen.
- Scores are % correct with paired bootstrap confidence intervals (10,000 resamples). On the held-out test they were stratified by cluster, Holm-corrected, and repeated with a second judge and a second seed.
- Every system shared the same answering model, embedder and answer prompt. Each competitor ran with its documented defaults and hooks; Mem0 is its best-configured arm per track.
- Versions: Mem0 2.2.1 (open source), Graphiti 0.31.2 with FalkorDB, Supermemory local 0.0.9. Plain vector search uses one index with 400-token chunks; per-department vector search uses one index per department plus a company index, with 200-token chunks.
- Development rows come from different studies and research versions. Compare within a row, not across rows.
VIINext
The loss on real history is the problem to solve, and it is the one we are working on first: getting large, unstructured history into the vault without losing what was said. When there is a new version, it gets a new sealed test, and we will publish that result here whichever way it goes.
Every number, with confidence intervals and footnotes, is on the Open Passport benchmarks page.