Hippocampus on LegalBench-RAG: 86% success rate

The test

LegalBench-RAG, by ZeroEntropy, asks for the clause that answers a question about a specific contract or policy: NDAs, merger agreements, commercial contracts, privacy policies. We ran its mini split: 776 questions over 72 documents that often look alike. It is one of the tests in our benchmark report, version 3.1, of 20 October 2026.

How to read the numbers

• Success rate: of the questions a perfect search would solve, the share each system solves. A perfect search here is the answer model handed each question's own evidence pages; the questions it still misses (a wrong reference answer, an ambiguous question, or one too hard even with the right pages) are left out.
• Answered right (all questions): every question counts, nothing filtered out. An answer is right when the judge finds it in line with the dataset's reference answer. Hippocampus 82%, Azure AI Search 75%.
• Evidence found: the share of the passages needed to answer that the search found, on average.
• For scale: with no documents, the answer model gets 28.2% right from memory. With the right pages, it gets 92.0% right.

Results

• Head to head against Hippocampus: Azure AI Search 102 : 45 (p < 10⁻⁵), Vertex AI Search 125 : 53 (p < 10⁻⁷), OpenAI File Search 134 : 51 (p < 10⁻⁹), OpenAI Direct Search 149 : 35 (p < 10⁻¹⁷). Each pair is the questions Hippocampus got right and the other system did not, against the reverse (two-sided sign test).

The reference is a passage rather than a short answer, so an answer worded differently from the clause can lose credit; about 3% of the questions (20 of 776) are disputed.

Vertex AI Search and quotation marks. Vertex AI Search reads double quotation marks as an exact-phrase operator, and LegalBench-RAG puts party names in quotes. Sent as written, 347 of the 373 quoted questions returned no passage, and Vertex AI Search answered 37% of all questions right. The other systems returned passages for every one. For Vertex AI Search alone we removed the quotation marks from the search query, which gives its figures in the table. The answer step saw every question as written.

Our misses

An automated audit checked the 141 questions Hippocampus did not answer right: 88 real misses, 4 ambiguous questions, 27 reference problems and 22 right but less complete than the reference. Real misses are 11% of all questions. The audit is an aid to reading the misses, not a human verdict.

Speed

Hippocampus is the most accurate here, not the fastest. At the median, a search takes 1.8 seconds in Hippocampus, 0.4 seconds in Azure AI Search, 0.3 seconds in Vertex AI Search and 1.7 seconds in OpenAI Direct Search. These times are from version 2.4 of the report and cover the search only.

On contracts the systems are closer together than on the financial filings, and the lead is still clear against each of them. When documents look alike, as agreements of the same kind do, the extra time buys a different result.

How we tested

Every system indexed the same files and made one search per question. The same answer model (GPT-5.4 mini) and the same judge (Claude Sonnet 5) handled every answer. The full settings and prompts are on the evidence page under the Legal tab.

The dataset

LegalBench-RAG (the mini split) is by ZeroEntropy, at github.com/zeroentropy-ai/legalbenchrag, under the MIT licence. We use it as published and report scores only: no question, answer or passage is reproduced here.