Benchmarks
All results reproducible from the benches/ directory. Protocol details in docs/benchmarks.
CausalEval — the causal memory benchmark
Typed DAGs are generated deterministically; conversations are narrated from the graph; gold answers are derived from graph structure — zero hand annotation, zero ambiguity.
| Capability | causal-memory | mem0 | What it tests |
|---|---|---|---|
| C7 Update | 100% | 80% | Supersede old belief after falsification (soft superseded_by annotation) |
| C3 Counterfactual | 95% | 80% | Choosing between alternatives with known outcomes |
| C2 Intervention | 75% | 40% | Forward prediction: "if X again, what happens?" |
| C4 Inhibition | 80% | 50% | Root-cause fix vs blast-radius limiter (prevented edges) |
| C1 Attribution | 85% | 90% | Backward causal chain → root cause |
| C5 Temporal-causal | 90% | 90% | Ordering on a causal chain |
| C6 Lesson transfer | 20% | 30% | Cross-task analogy via meta edges (open limitation) |
| Overall | 78% | 65% | CausalEval v13 · 140 questions, 20 graphs, same LLM, same judge |
Fact-recall benchmarks
On traditional fact-recall suites causal-memory is competitive but does not beat mem0 — fact recall is mem0's specialty, not where causal-memory adds value.
| Benchmark | causal-memory | mem0 | Note |
|---|---|---|---|
| LoCoMo (strict judge) | 79.1% | 91.6% | mem0's home turf |
| LongMemEval-S | 76.4% @ 11.5K tok/q | 94.4% (official) | single-model stack vs platform stack |
| Memora MPA | 67.4% | 71.8% | −4.4pp |
| Compaction survival | 100% | 45% | External table = immune to compaction |
| Agent repeat-mistake | 33% | 67% | −34pp on trap-world |
Capability tests
322 workspace tests covering capabilities no fact store (mem0, Zep, Letta) offers.
| Capability | What it proves |
|---|---|
| Prevented-edge warning | prevented edge spreads −0.3 activation (GABA analogue) |
| Trace-cause attribution | Backward CSR traversal finds root cause |
| Multi-hop causal chain | Forward K-hop spreading reaches 2–3 hop outcomes |
| Inhibitory filtering | Prevented outcomes appear as negative, not false positives |
| Intervention comparison | Same outcome: +0.9 for "skip tests", −0.3 for "add tests" |
| SWR consolidation | LTP strengthens replayed edges, LTD weakens unvisited, GC forgets dormant |
| Q-value dynamics | Good decisions rank higher; Bellman propagates to parents |
| Novelty entropy | Diverse experience triggers consolidation; uniform does not |
| Meta-edge mining | Cross-session pattern discovery (similar_to / repeated) |
| Hebbian co-occurrence | Repeated co-activation strengthens connection |