Verification is our product, so our own numbers are held to the standard we sell: each figure below names its dataset, sample size, model, protocol, and the machine-readable result file that produced it. All evaluations are self-run; no external party has audited them yet, and we say so.
Last updated July 21, 2026 · Datasets and evaluation code: github.com/andremir/car-retrieval · CAR paper: arXiv 2604.14488
Protocol, identical across benchmarks: each item asks whether a legacy authority is still controlling (a superseded accounting standard, an overruled precedent, a recalled drug). PortMem condition: Claude Haiku 4.5 answers from PortMem-retrieved passages (currency- and authority-aware retrieval). Baseline condition: the same Claude Haiku 4.5 answers from its own knowledge with no documents. Answers are graded by an LLM judge on one criterion: does the answer recognize the supersession/recall and name the successor. Temperature 0; one run per cell.
| Benchmark | n | With PortMem | Same model, no documents | Same model, full corpus pasted | Artifacts |
|---|---|---|---|---|---|
| FASB ASC supersessions 20 real accounting-standard transitions |
20 | 100% | 95% | 100% | fasb_vs_llm_legal_kb_track_a_only.json (2026-05-08) · fasb_knowledge_only_baseline.json · fasb_llm_alone_baseline.json (2026-07-21) |
| SCOTUS overruling pairs "Is this precedent still good law?" |
100 | 99% | 48% | 96% | scotus_vs_llm_legal_kb_track_a_only.json (2026-05) · scotus_knowledge_only_baseline.json · scotus_llm_alone_baseline.json (2026-07-21) |
| FDA recall lookups "Is this drug currently approved?" |
100 | 99% | 29% | 80% | portmem_vs_llm_claude_haiku_4_5_track_a_only.json (2026-05-09) · fda_knowledge_only_baseline.json · fda_llm_alone_baseline.json (2026-07-21) |
Read the full-corpus column honestly: these benchmark corpora are small and curated, so pasting everything into the context window works well on them. That column is not the product case. The product case is real document estates that do not fit a context window, plus what pasting can never produce: a citation to the controlling version, a claim-level verdict, and a signed record. On the currency-ranking ablation, relevance ranking alone put the controlling version first in 0 of 8 topics; PortMem's currency capability alone scored 7 of 8, and the shipped combination 8 of 8.
The legal corpus was audited against primary sources and relabeled, with the audit published alongside the paper.
Until this date, the benchmark tiles on our homepage showed baselines of 65%, 7%, and 65% for "a workhorse frontier model alone." During an internal audit we could not trace two of those three figures to a reproducible artifact under the stated condition, so we re-ran the baselines under two explicit, named conditions (no-documents and full-corpus-paste) and replaced the displayed figures with the measured results above.
The correction cuts both ways: the FASB baseline got harder for us (95%, not 65% — frontier models know the famous accounting transitions), and the FDA baseline got easier (29%, not 65%). We publish the change because a verification company that quietly patches its own numbers has no product. This is the same discipline recorded in our evidence log: when we could not reproduce our best retrieval benchmark blind, we demoted it and printed why.