Transparency archive for benchmark campaigns run against Mnemosyne OS’s memory engine — methodology, honest caveats, and the raw run logs behind every published number.
📊 Live results page → — the numbers, the per-question ledger, and how to recompute them yourself.
Why a separate repo: a benchmark claim deserves scrutiny the product README isn’t the place for. Everything here is real run output — not curated screenshots — so the numbers can be checked, not just trusted.
Don’t take the headline on faith — re-derive it. The verification kit ships the exact grader, the per-question verdicts behind each score, and a one-command tool that recomputes the accuracy from those verdicts — no memory engine, no network, no dependencies:
cd verification-kit
node verify.js # recompute every score from its per-question rows
node scoring.js --selftest # audit the grader on real cases
verify.js proves each advertised score is the exact sum of the published
per-question rows — no hidden questions, no arithmetic massaging. Every ledger
recomputes in full: baseline (64.6%), engine multi-session (5/8), local-sovereign
(50%), and all four arms of the August campaign — vector-only and fused, each read
by both the strict and the flexible judge (29/48, 37/48, 34/48, 39/48).
The current headline is 77.1% (37/48), August 2026, under a strict judge —
one 48-question run, replayed a second time and agreeing verdict for verdict, with
the retrieval gain confirmed on a 48-question holdout never seen during development.
Under July’s flexible judge, the same answers measure 81.3% (39/48, same
both-runs rule) — and verify.js recomputes that one too, from its own published
ledger. Which judge graded a number changes what it means, so every score here
names its judge, and both leniencies of the August campaign ship as ledgers rather
than as an assertion.
The two graders disagree about more than the headline: under the strict judge the channel gains +9/−1 questions, under the flexible one +7/−2. The flexible reading is the weaker evidence, and it is published at full strength rather than rounded away.
July’s 72.9% is a different kind of number, and verify.js says so out loud. It was
never measured in a single 48-question engine run — only the multi-session
category was re-run with the engine, and the other 40 rows are carried from the
baseline ledger. The tool recomputes and prints that composition
(30/40 carried + 5/8 measured = 35/48) on every run, so the number cannot
quietly detach from how it was built. It is therefore a lower bound: the 40
carried questions were never retried, so a full re-run can only raise it.
Details in the kit’s RESULTS.md and METHODOLOGY.md §5.
This is why 72.9% and 77.1% are published side by side rather than as a single progression: one is a composed lower bound under a flexible judge, the other a measured run under a strict one. July’s campaign stays exactly as it was published, DOI-pinned — a number improved on is not a number withdrawn.
➡️ Rendered results page:
mnemosyne-os.github.io/MnemosyneOS—benchmarks/verification-kit
— that’s the live site; the files in verification-kit/
are the source behind it.
| Campaign | Headline | |
|---|---|---|
| LongMemEval-M full-haystack (2026-07) | 64.6 % → 72.9 %, multi-session recall 1/8 → 5/8 | 16 raw run logs |
| Lexical channel — hybrid retrieval, fully local (2026-08) | strict judge 29/48 → 37/48 (reproduced ×2, p = 0.0215), holdout on 48 unseen questions +4/−0 sessions, zero regressions | 11 raw run files |
LongMemEval is a public,
independent long-term-memory benchmark. Its full-haystack variant surrounds
every question’s evidence with ~480 distractor sessions from other personas —
the closest published setup to a real, lived-in memory vault. Most reported
numbers (including the original paper’s) use the easier -S variant instead.
Mnemosyne’s numbers above are on the harder one.
These rules are written down as an enforceable contract in AGENTS.md
— the ledger format, the replay discipline, the rule that a composed figure may
never be presented as a measured one, and the checklist that has to pass before
anything here is updated. It’s addressed to whoever (or whatever) publishes the
next number, and it’s public for the same reason the logs are.
Be clear about what this repo does and doesn’t let you do. It lets you audit the scoring — the grader, the per-question verdicts, the arithmetic, the replay discipline, and every row traced back to the public dataset. It does not let you reproduce the retrieval: the engine that produced the answers is closed, so you cannot re-run generation and get these logs back. That is a real limit and we’d rather state it than let the word “reproducible” imply otherwise.
The internal harness scripts aren’t published here — they’re coupled to Mnemosyne’s core engine, not standalone tools. They’re available on motivated request: open an issue on this repo explaining what you’d like to verify or extend.
The benchmark datasets themselves are public (linked in each campaign’s summary) — the logs here are enough to check the scoring and methodology against them independently.
The campaign write-ups, in plain language, on the product site:
verification-kit/verify.js, scoring.js, index.html:
MIT, so you can fork the grader and check it against your own
results without a content licence getting in the way.