Most memory benchmarks ask you to take a headline on faith. This one ships the grader, the per-question verdicts, and a one-command tool that re-derives the score from those verdicts — with no engine and no network. What stays private is the memory engine that produces the answers, never anything you need to check the scoring.
Latest · the rerun of 2026-09-30, protocol published before it ran
Hybrid mode: embeddings and search run on the machine, and a cloud model, gemini-3.8-flash, writes the answers. Every answer, every raw judge reply, the human audit and a verify.js are in longmemeval-rerun-2026-10/. The reader, the engine and the strict judge's model all changed since August, so the figures below do not chain into these.
Earlier campaigns, July and August 2026
LongMemEval-M · full-haystack variant (~480 distractor sessions per question) · answers gemini-2.5-pro / Qwen2.5-3B · LLM judge, flexible in July and strict in August — each score below says which one graded it · e5 embeddings
verify.js reads the published per-question ledgers, recomputes accuracy and the per-category breakdown, and asserts they equal the claimed headline. It proves the advertised number is the exact sum of the published rows — no hidden questions, no arithmetic massaging.
# recompute every published score from its per-question rows $ node verify.js ✓ Baseline (retrieval only, no consolidation engine) recomputed from 48 published rows: 31/48 = 64.6% claimed headline: 31/48 = 64.6% match: total ok · hits ok · accuracy ok ✓ Full engine — multi-session category (8q) recomputed from 8 published rows: 5/8 = 62.5% replay rule: 2 first-run HITs did NOT reproduce → scored MISS ✓ COMPOSED HEADLINE — full engine, full-haystack This number is COMPOSED from two ledgers, not measured in one 48q run: 30/40 carried unchanged from the baseline (never re-run with the engine) 5/8 measured with the engine (multi-session only) 35/48 = 72.9% recomputed claimed: 35/48 = 72.9% → ok ✓ ALL LEDGERS CONSISTENT — every headline, including the composed one, is the exact sum of the published rows. # and audit the grader itself against the public ground truth $ node scoring.js --selftest ✓ self-test: 22/22 expected verdicts reproduced
Trace any row yourself. Every question id below is the official LongMemEval id — look it up in the public dataset to see the question, the haystack, and the ground-truth answer.
A second, fully local retrieval channel — Okapi BM25 over a persistent inverted index inside the vault, merged with the dense ranking by Reciprocal Rank Fusion (literature defaults, deliberately untuned) — measured under a strict judge, with a 48-question holdout never seen during development.
| Measurement | Vector only | + lexical channel |
|---|---|---|
| Evidence sessions retrieved (deterministic, no LLM) | 38/48 | 41/48 |
| Answer-bearing chunk served (35 tracked) | 25/35 | 30/35 |
| Holdout — 48 questions never seen in development | — | +4/−0 sessions · +2/−0 chunks |
| End-to-end, strict judge, both-runs replay rule | 29/48 · 60.4% (one run) | 35/48 · 72.9% (+9/−3 paired, p = 0.146) |
| The same answers, flexible judge (July's grader), same rule | 34/48 · 70.8% (one run) | 37/48 · 77.1% (+7/−4 paired, p = 0.549) |
Audit it: raw runs and the reading key live in lexical-2026-08/; the ledgers are lexical-baseline-strict-48q.jsonl, lexical-fusion-strict-48q.jsonl and their flexible twins — the same answers under both graders, so the leniency difference is something you check rather than something we assert. All four are mechanically extracted (extract-ledgers.mjs), both runs named on every row. node verify.js recomputes every cell above, including the holdout, and fails on any mismatch. The development sample runs ~13 points easier than its parent set (why the holdout exists), the judge's noise floor is ≈2.6 verdicts/48, and no product comparison is claimed — both arms are our own montages under our own protocol. The holdout measures retrieval only. The answer-side gain is not statistically significant.
Corrected on 2026-09-28. The end-to-end rows read 37/48 and 39/48 until an external audit by Julien Gelee. A human reading overturned two judge verdicts: gpt4_76048e76 (expected "bike", the answer says "car first") and 6ade9755 (run 2 names the right yoga studio, then says the user is still looking for one). Both are listed in human-audit.json, shown on their ledger rows, and printed by verify.js. Full record: ERRATUM.md.
Every question, every verdict. This is the full record behind the 64.6%. Filter it, scan it, then run verify.js and watch the same 31/48 fall out.
| Question id | Category | Verdict | Model answer → expected |
|---|
Answer excerpts are trimmed for reading; the verdicts they carry are the real graded verdicts. Full untruncated transcripts are available on request.
The engine's 8 multi-session rows are a separate ledger: results/engine-multisession-8q.jsonl. Each row names both runs behind its verdict, carries a baseline_correct field so you can see which questions the engine actually recovered, and keeps the two discarded first-run HITs visible with their reason. Those 8 rows plus the 40 carried baseline rows above are exactly what verify.js composes into 35/48.
The consolidation engine earns its keep exactly where memory gets hard: multi-session aggregation — questions whose answer is scattered across many conversations buried in hundreds of distractors.
How July's 72.9% is built. It is composed, not measured in one run. Only the multi-session category (8 questions) was re-run with the engine; the other 40 rows are carried verbatim from the baseline ledger. There has never been a single 48-question run of the full engine, and verify.js prints that composition every time it runs. This page also called it a lower bound until 2026-09-28. That claim is withdrawn: nothing showed a full re-run can only raise it, and an August full-engine run of the same 48 questions under the same flexible judge scored 34/48. Also worth stating: 1 of the 5 multi-session HITs (e831120c) was already a baseline HIT, so the engine's net recovery is 4 questions, not 5.
What we discarded, and why. A HIT counts only if it reproduces on a second run. Two first-run HITs did not, and both are published in the ledger with their discarded verdict intact rather than deleted: gpt4_59c863d7 (model kits) — judge noise: the engine gave the same answer ("6 kits", ground truth "five") in both runs, and only the first judge called it a HIT. aae3761f (road trips) — answer variance: the first run was scored HIT, but its logged answer is cut at ~200 characters, before any total; the replay found only two of the three trips and answered "11 hours". Both are scored MISS. You may disagree with discarding them — the data to argue the other way is published.
Full engine on oracle-evidence vaults (no distractors) — the easy ceiling, not the headline.
Literature GPT-4o full-context — but on the easier -S variant, not full-haystack.
Fully local, 3B model, zero cloud — the price of sovereignty, reported plainly.
The remaining full-engine misses are decomposed, not hand-waved: two are benchmark-structural (other personas' first-person text in the shared haystack), one a write/read coverage trade-off.