Most memory benchmarks ask you to take a headline on faith. This one ships the grader, the per-question verdicts, and a one-command tool that re-derives the score from those verdicts — with no engine and no network. What stays private is the memory engine that produces the answers, never anything you need to check the scoring.
LongMemEval-M · full-haystack variant (~480 distractor sessions per question) · answers gemini-2.5-pro / Qwen2.5-3B · LLM judge, flexible in July and strict in August — each score below says which one graded it · e5 embeddings
verify.js reads the published per-question ledgers, recomputes accuracy and the per-category breakdown, and asserts they equal the claimed headline. It proves the advertised number is the exact sum of the published rows — no hidden questions, no arithmetic massaging.
# recompute every published score from its per-question rows $ node verify.js ✓ Baseline (retrieval only, no consolidation engine) recomputed from 48 published rows: 31/48 = 64.6% claimed headline: 31/48 = 64.6% match: total ok · hits ok · accuracy ok ✓ Full engine — multi-session category (8q) recomputed from 8 published rows: 5/8 = 62.5% replay rule: 2 first-run HITs did NOT reproduce → scored MISS ✓ COMPOSED HEADLINE — full engine, full-haystack This number is COMPOSED from two ledgers, not measured in one 48q run: 30/40 carried unchanged from the baseline (never re-run with the engine) 5/8 measured with the engine (multi-session only) 35/48 = 72.9% recomputed claimed: 35/48 = 72.9% → ok ✓ ALL LEDGERS CONSISTENT — every headline, including the composed one, is the exact sum of the published rows. # and audit the grader itself against the public ground truth $ node scoring.js --selftest ✓ self-test: 6/6 expected verdicts reproduced
Trace any row yourself. Every question id below is the official LongMemEval id — look it up in the public dataset to see the question, the haystack, and the ground-truth answer.
A second, fully local retrieval channel — Okapi BM25 over a persistent inverted index inside the vault, merged with the dense ranking by Reciprocal Rank Fusion (literature defaults, deliberately untuned) — measured under a strict judge, with a 48-question holdout never seen during development.
| Measurement | Vector only | + lexical channel |
|---|---|---|
| Evidence sessions retrieved (deterministic, no LLM) | 38/48 | 41/48 |
| Answer-bearing chunk served (35 tracked) | 25/35 | 30/35 |
| Holdout — 48 questions never seen in development | — | +4/−0 sessions · +2/−0 chunks |
| End-to-end, strict judge, both-runs replay rule | 29/48 · 60.4% | 37/48 · 77.1% (+9/−1 paired, p = 0.0215) |
| The same answers, flexible judge (July's grader), same rule | 34/48 · 70.8% | 39/48 · 81.3% (+7/−2 paired) |
Audit it: raw runs and the reading key live in lexical-2026-08/; the ledgers are lexical-baseline-strict-48q.jsonl, lexical-fusion-strict-48q.jsonl and their flexible twins — the same answers under both graders, so the leniency difference is something you check rather than something we assert. All four are mechanically extracted (extract-ledgers.mjs), both runs named on every row. node verify.js recomputes every cell above, including the holdout, and fails on any mismatch. The development sample runs ~13 points easier than its parent set (why the holdout exists), the judge's noise floor is ≈2.6 verdicts/48, and no product comparison is claimed — both arms are our own montages under our own protocol.
Every question, every verdict. This is the full record behind the 64.6%. Filter it, scan it, then run verify.js and watch the same 31/48 fall out.
| Question id | Category | Verdict | Model answer → expected |
|---|
Answer excerpts are trimmed for reading; the verdicts they carry are the real graded verdicts. Full untruncated transcripts are available on request.
The engine's 8 multi-session rows are a separate ledger: results/engine-multisession-8q.jsonl. Each row names both runs behind its verdict, carries a baseline_correct field so you can see which questions the engine actually recovered, and keeps the two discarded first-run HITs visible with their reason. Those 8 rows plus the 40 carried baseline rows above are exactly what verify.js composes into 35/48.
The consolidation engine earns its keep exactly where memory gets hard: multi-session aggregation — questions whose answer is scattered across many conversations buried in hundreds of distractors.
How 72.9% is built — two separate caveats, kept separate. (1) It is composed, not measured in one run. Only the multi-session category (8 questions) was re-run with the engine; the other 40 rows are carried verbatim from the baseline ledger. There has never been a single 48-question run of the full engine, and verify.js prints that composition every time it runs. (2) It is a lower bound — those 40 carried questions were never retried with consolidation, so a full re-run can only raise the figure. Also worth stating: 1 of the 5 multi-session HITs (e831120c) was already a baseline HIT, so the engine's net recovery is 4 questions, not 5.
What we discarded, and why. A HIT counts only if it reproduces on a second run. Two first-run HITs did not, and both are published in the ledger with their discarded verdict intact rather than deleted: gpt4_59c863d7 (model kits) — judge noise: the engine gave the same answer ("6 kits", ground truth "five") in both runs, and only the first judge called it a HIT. aae3761f (road trips) — answer variance: the first run answered "15 hours" and was correctly scored HIT, the replay found only two of the three trips and answered "11 hours". Both are scored MISS. You may disagree with discarding them — the data to argue the other way is published.
Full engine on oracle-evidence vaults (no distractors) — the easy ceiling, not the headline.
Literature GPT-4o full-context — but on the easier -S variant, not full-haystack.
Fully local, 3B model, zero cloud — the price of sovereignty, reported plainly.
The remaining full-engine misses are decomposed, not hand-waved: two are benchmark-structural (other personas' first-person text in the shared haystack), one a write/read coverage trade-off.