Hippocampus on SECQUE: 81% success rate

The test

SECQUE, by Ben Yoash and colleagues, asks 565 expert questions that compare and compute across the financial statements of 45 SEC filings: ratios, trends, risk. An answer often needs several passages and a calculation. It is one of the tests in our benchmark report, version 3.1, of 20 October 2026.

How to read the numbers

• Success rate: of the questions a perfect search would solve, the share each system solves. A perfect search here is the answer model handed each question's own evidence pages; the questions it still misses (a wrong reference answer, an ambiguous question, or one too hard even with the right pages) are left out.
• Answered right (all questions): every question counts, nothing filtered out. An answer is right when the judge finds it in line with the dataset's reference answer. Hippocampus 67%, Azure AI Search 48%.
• Evidence found: the share of the passages needed to answer that the search found, on average.
• For scale: with no documents, the answer model gets 26.4% right from memory. With the right pages, it gets 71.7% right.

Results

• Head to head against Hippocampus: Azure AI Search 134 : 23 (p < 10⁻¹⁹), OpenAI File Search 186 : 16 (p < 10⁻³⁷), Vertex AI Search 196 : 23 (p < 10⁻³⁴), OpenAI Direct Search 224 : 13 (p < 10⁻⁵⁰). Each pair is the questions Hippocampus got right and the other system did not, against the reverse (two-sided sign test).

Reference answers are long, so an answer can be right and still miss one of their points; about 13% of the questions (71 of 565) are disputed.

Our misses

An automated audit checked the 185 questions Hippocampus did not answer right: 64 real misses, 56 ambiguous questions, 35 reference problems and 30 right but less complete than the reference. Real misses are 11% of all questions. The audit is an aid to reading the misses, not a human verdict.

Speed

Hippocampus is the most accurate here, not the fastest. At the median, a search takes 1.8 seconds in Hippocampus, 0.4 seconds in Azure AI Search, 0.3 seconds in Vertex AI Search and 1.7 seconds in OpenAI Direct Search. These times are from version 2.4 of the report and cover the search only.

For a plain answer in one clearly distinct document, a fast cloud search is often good enough. When an answer needs several passages, as on SECQUE, the extra time buys a different result: 81% against 58% for the best cloud service and 46% for OpenAI File Search.

How we tested

Every system indexed the same files and made one search per question. The same answer model (GPT-5.4 mini) and the same judge (Claude Sonnet 5) handled every answer. The full settings and prompts are on the evidence page under the Finance tab.

The dataset

SECQUE is by Ben Yoash et al., at huggingface.co/datasets/nogabenyoash/SecQue, under the MIT licence. We use it as published and report scores only: no question, answer or passage is reproduced here.