BENCHMARK
CausalEval: the causal graph is the answer key
Fact-recall suites cannot see inhibition, intervention or belief revision. CausalEval generates a typed causal graph, narrates conversations from it, and derives every gold answer from the graph.
Updated · English
Why a new benchmark
LoCoMo, LongMemEval and Memora ask whether a system can recall a fact from a long conversation. They are good at that, and mem0 leads them. None of them can ask whether a system knows that one action prevented another, or that a belief was superseded last week, because those questions have no fact-shaped answer.
CausalEval starts from the opposite direction. A typed directed acyclic graph is generated first — nodes are people, actions and events, edges are caused, enabled or prevented. Conversations are then written from the graph, so the graph itself is the ground truth. There is no annotator judgement to disagree with and no ambiguity about which answer is correct.
Protocol
v13 (soft supersession) · 140 questions over 20 graphs · same LLM, same judge
| Version | v13 (soft supersession) |
|---|---|
| Questions | 140 |
| Graphs | 20 |
| Baseline | v12: 70 questions / 10 graphs · mem0 comparison ran on the same 70q protocol |
| Judge | same LLM, same judge |
| Gold answers | Derived from graph structure — zero hand annotation |
Seven capability classes
| Class | Question shape | Gold source | Capability |
|---|---|---|---|
| C1 | Why did this outcome happen? | The backward causal chain from the root decision to the outcome | trace_cause / trace_cause_chain |
| C2 | If this decision is made again, what will happen? | The forward outcome implied by caused/enabled/prevented edges | intervention_query |
| C3 | Should the agent do X or Y, given known outcomes? | The option whose recorded outcome is positive or safe | counterfactual_query |
| C4 | What stopped this outcome from being worse? | The prevented edges targeting the outcome | prevented negative spread |
| C5 | Which happened first, A or B? | The construction order of events on the causal chain | temporal-causal reasoning |
| C6 | Facing a similar situation in another task, what should be avoided? | The failure lesson reachable through meta edges | meta-edge mining |
| C7 | What does the agent now believe, after the old belief was falsified? | The corrected statement; the superseded edge stays retrievable | soft supersession |
Results
Source: causal-memory README v0.10.0 · CausalEval v13, read 2026-10-21. The baseline column is the mem0 run on the same 70-question protocol.
| Capability | causal-memory | mem0 (70q baseline) | What it tests |
|---|---|---|---|
| C7 Update | 100% | 80% | Supersede an old belief after it is falsified, and keep the old edge retrievable |
| C3 Counterfactual | 95% | 80% | Choose between alternatives with known outcomes |
| C2 Intervention | 75% | 40% | Forward prediction: if X happens again, what follows? |
| C4 Inhibition | 80% | 50% | Separate the root-cause fix from the blast-radius limiter |
| C1 Attribution | 85% | 90% | Walk a backward causal chain to the root cause |
| C5 Temporal-causal | 90% | 90% | Order events on a causal chain |
| C6 Lesson transfer | 20% | 30% | Cross-task analogy through meta edges — still an open limitation |
| Overall | 78% | 65% | All C1-C7 classes weighted equally |
C7 update moved from 50% (v12) to 100% (v13) once superseded edges were annotated instead of hidden, and it held at doubled sample size. C6 lesson transfer is still 20% against mem0's 30% — that gap is open.
The limitation we are not hiding
C6 lesson transfer: 20% vs 30%
C6 asks the system to carry a lesson from one task to an isomorphic situation in a different task category. It requires meta edges that connect patterns across sessions, and it is the one class where causal-memory loses to the fact-store baseline: 20% against 30%. Cross-task analogy is the weakest capability in the suite and it is not fixed.
There are three smaller caveats. C1, C4 and C5 dipped relative to the v12 run; those dips are within re-distillation variance and new-graph difficulty, and v12 and v13 do not share a distilled corpus, so they should not be read as regressions. The suite is 140 questions over 20 graphs — enough to separate the systems on most classes, not a large-scale evaluation. And narration and judging both use an LLM, which adds variance that a deterministic harness would not have.
Fact recall: the part where causal-memory loses
CausalEval is not the whole story. On the traditional suites below, mem0 leads and this page reports it.
| Capability | causal-memory | mem0 / baseline | What it tests |
|---|---|---|---|
| LoCoMo (strict judge) | 79.1% | 91.6% | mem0's home turf — atomic fact extraction and recall |
| LongMemEval-S | 76.4% @ 11.5K tok/q | 94.4% official · 73.8% independent repro | Single-model stack vs a platform stack; not a like-for-like number |
| Memora MPA | 67.4% | 71.8% | −4.4pp; fact-heavy multi-session recall |
| Compaction survival | 100% | 45% | Recall after 5x context compaction; the causal table lives outside the window |
| Agent repeat-mistake | 33% | 67% | −34pp on trap-world: how often the agent repeats a known-bad action |
LoCoMo 79.1% vs 91.6% is the cleanest loss: same suite, strict judge, mem0's home turf. Compaction survival (100% vs 45%) is the cleanest win, and it is a direct consequence of the design: the causal table lives outside the context window, so compaction cannot rewrite it.
Reproduce it
Generation is deterministic and seeded. Narration and judging call an LLM, so those steps need an API key.
# 1. deterministic graphs + graph-derived questions
cargo run --release --bin causal-memory-causal-eval -- \
generate --graphs 20 --out eval-data
# 2. narrate each graph into multi-session dialogue (LLM)
cargo run --release --bin causal-memory-causal-eval -- \
narrate --data eval-data
# 3. ingest, retrieve and judge per capability
cargo run --release --bin causal-memory-causal-eval -- \
run --data eval-dataThe protocol document and the per-class question templates live in the repository under docs/benchmarks/causal-eval-2026.md. If you run the mem0 baseline, use the 70-question protocol so the comparison is like-for-like.
How to read these numbers
A benchmark is an argument, not a verdict. CausalEval is built to make one specific argument checkable: that typed causal structure is a different capability from fact recall, and that a memory system can be good at one and mediocre at the other. The per-capability columns exist so that a reader can disagree with the aggregate.
- Compare per class, not only the overall score — the overall average hides C6 and the C1/C4/C5 variance.
- Do not compare the 140q v13 column to a 70q baseline as if it were the same experiment.
- Treat the fact-recall table as the counter-argument; it is on the page for that reason.
- Reproduce before quoting: the harness and the protocol both ship with the repository.
CausalEval questions
- What is CausalEval?
- A benchmark for the memory capabilities that fact-recall suites cannot measure: typed causal reasoning, inhibition, intervention prediction, cross-task lesson transfer and belief revision. Its defining property is that the causal graph is the answer key: graphs are generated deterministically, conversations are narrated from the graph, and every gold answer is derived from graph structure rather than hand annotation.
- Why not just use LoCoMo or LongMemEval?
- Those suites measure fact recall, which is mem0's strength and not where causal memory adds value. On LoCoMo causal-memory scores 79.1% against mem0's 91.6% — a real loss, reported here rather than hidden. CausalEval exists because C2, C4, C6 and C7 are invisible on a fact-recall suite; it is a complement, not a replacement.
- Is the mem0 comparison on this page fair?
- It is a like-for-like comparison on the 70-question protocol that both systems ran, not on the 140-question v13 set. The v13 column is causal-memory only. We report it that way instead of presenting 140q numbers next to 70q numbers as if they were the same experiment.
- How do I reproduce the results?
- The harness is a binary in the repository: causal-memory-causal-eval with generate, narrate and run subcommands. Generation is deterministic and seeded; narration and judging need an LLM API key. The exact commands are on this page, and the protocol document ships in the repository under docs/benchmarks.
- What is the biggest limitation?
- C6 lesson transfer: 20% against mem0's 30%. Cross-task analogy through meta edges is the weakest capability in the suite and it is not fixed. The second limitation is scale — 140 questions over 20 graphs is enough to separate the systems on most classes, but it is not a large-scale evaluation.
Related
Check the number, then check the protocol
The harness ships with the repository. Generate the graphs, run the baseline, and compare the per-capability table with the one on this page.