LongMemEval-M · full-haystack variant (~480 distractor sessions per question) · answers gemini-2.5-pro / Qwen2.5-3B · flexible LLM judge · e5 embeddings
verify.js reads the published per-question ledgers, recomputes accuracy and the per-category breakdown, and asserts they equal the claimed headline. It proves the advertised number is the exact sum of the published rows — no hidden questions, no arithmetic massaging.
# recompute every published score from its per-question rows $ node verify.js ✓ Baseline (retrieval only, no consolidation engine) recomputed from 48 published rows: 31/48 = 64.6% claimed headline: 31/48 = 64.6% match: total ok · hits ok · accuracy ok ✓ Full engine — multi-session category (8q) recomputed from 8 published rows: 5/8 = 62.5% replay rule: 2 first-run HITs did NOT reproduce → scored MISS ✓ COMPOSED HEADLINE — full engine, full-haystack This number is COMPOSED from two ledgers, not measured in one 48q run: 30/40 carried unchanged from the baseline (never re-run with the engine) 5/8 measured with the engine (multi-session only) 35/48 = 72.9% recomputed claimed: 35/48 = 72.9% → ok ✓ ALL LEDGERS CONSISTENT — every headline, including the composed one, is the exact sum of the published rows. # and audit the grader itself against the public ground truth $ node scoring.js --selftest ✓ self-test: 6/6 expected verdicts reproduced
Trace any row yourself. Every question id below is the official LongMemEval id — look it up in the public dataset to see the question, the haystack, and the ground-truth answer.
Every question, every verdict. This is the full record behind the 64.6%. Filter it, scan it, then run verify.js and watch the same 31/48 fall out.
| Question id | Category | Verdict | Model answer → expected |
|---|
Answer excerpts are trimmed for reading; the verdicts they carry are the real graded verdicts. Full untruncated transcripts are available on request.
The engine's 8 multi-session rows are a separate ledger: results/engine-multisession-8q.jsonl. Each row names both runs behind its verdict, carries a baseline_correct field so you can see which questions the engine actually recovered, and keeps the two discarded first-run HITs visible with their reason. Those 8 rows plus the 40 carried baseline rows above are exactly what verify.js composes into 35/48.
The consolidation engine earns its keep exactly where memory gets hard: multi-session aggregation — questions whose answer is scattered across many conversations buried in hundreds of distractors.
How 72.9% is built — two separate caveats, kept separate. (1) It is composed, not measured in one run. Only the multi-session category (8 questions) was re-run with the engine; the other 40 rows are carried verbatim from the baseline ledger. There has never been a single 48-question run of the full engine, and verify.js prints that composition every time it runs. (2) It is a lower bound — those 40 carried questions were never retried with consolidation, so a full re-run can only raise the figure. Also worth stating: 1 of the 5 multi-session HITs (e831120c) was already a baseline HIT, so the engine's net recovery is 4 questions, not 5.
What we discarded, and why. A HIT counts only if it reproduces on a second run. Two first-run HITs did not, and both are published in the ledger with their discarded verdict intact rather than deleted: gpt4_59c863d7 (model kits) — judge noise: the engine gave the same answer ("6 kits", ground truth "five") in both runs, and only the first judge called it a HIT. aae3761f (road trips) — answer variance: the first run answered "15 hours" and was correctly scored HIT, the replay found only two of the three trips and answered "11 hours". Both are scored MISS. You may disagree with discarding them — the data to argue the other way is published.
Full engine on oracle-evidence vaults (no distractors) — the easy ceiling, not the headline.
Literature GPT-4o full-context — but on the easier -S variant, not full-haystack.
Fully local, 3B model, zero cloud — the price of sovereignty, reported plainly.
The remaining full-engine misses are decomposed, not hand-waved: two are benchmark-structural (other personas' first-person text in the shared haystack), one a write/read coverage trade-off.