See for yourself.
Public benchmarks of financial filings and contracts, and our FDA benchmark. Toggle between them and check the evidence yourself.
FinanceBench and LegalBench-RAG, tested on 20 October 2026, and our FDA benchmark. In the public benchmarks each system made one search per question, with the same answer model (GPT-5.4 mini, low reasoning) and the same judge (Claude Sonnet 5, majority of three votes). Search time was measured in version 2.4 of our benchmark report.
- 92% success rate on FinanceBench Of the questions a perfect search would solve, Hippocampus solves 92%. Next best: 77%.
- 86% success rate on LegalBench-RAG Of the questions a perfect search would solve, Hippocampus solves 86%. Next best: 78%.
- 1.8 s Hippocampus median search time Vertex AI Search 0.3 s, Azure AI Search 0.4 s, both less accurate.
Head to head
FinanceBench: success rate by system
150 analyst questions on 84 US company filings, from Patronus AI. Each bar is the share of answerable questions the system gets right with its own search.
Read the full reportHead to head
LegalBench-RAG: success rate by system
776 questions on 72 contracts and privacy policies, from ZeroEntropy. Each bar is the share of answerable questions the system gets right with its own search.
Read the full reportHead to head
Healthcare: answers right by system
Every system answered through GPT-5.4 mini, the same small model.
Read the full reportHealthcare
But with a stronger model, the result improves even more.
- 100% retrieval accuracy Claude Opus 5 answering. A correct source document reached for every answerable question, every repetition.
- 97% correct Claude Opus 5 answering. Two full runs of the question set (47 of 48 and 46 of 48), including the control question declined in all six repetitions.
- 6/6 control question declined Claude Opus 5 answering. Asked something the documents cannot answer, it declined the control question in all six repetitions.
How we tested
The same test for every system
On Finance and Legal, every system indexed the same documents and made one search per question. The same answer model (GPT-5.4 mini, low reasoning) wrote every answer, and the same judge (Claude Sonnet 5, three votes) graded it. On Healthcare, GPT-5.4 mini wrote every answer in the bars, GPT-5 graded them (three votes), and Hippocampus could search up to 12 times.
Pick Finance, Legal or Healthcare to see how to reproduce that benchmark.
How to reproduce
Repeat the finance results
Download the datasets, set up each system as below, and grade with the prompts.
Datasets
- FinanceBench (open-source 150), Patronus AI, github.com/patronus-ai/financebench, licence CC BY-NC 4.0
- SECQUE, Ben Yoash et al., huggingface.co/datasets/nogabenyoash/SecQue, licence MIT
Used as published. This page reproduces no question, answer or passage from them.
How each system was set up
- OpenAI File Search. Vector store with automatic chunking; the file-search tool, 10 results, one search.
- OpenAI Direct Search. The same vector store, searched once with the question, 10 results.
- Azure AI Search. Import-wizard vectorization, 2,000-character chunks with 500 overlap, text-embedding-3-large; one hybrid query with the semantic ranker, top 10.
- Vertex AI Search. Standard edition, EU; layout parser, 500-token chunks with ancestor headings; one search, default ranking, query expansion off, top 10.
- Hippocampus. Default settings, nothing tuned per test; one search per question through the search API, search release of 18 October 2026.
How answers were graded
- Same documents. Every system indexed the identical files, pinned by checksum.
- One search per question. The question as written; only OpenAI File Search lets the model write its query.
- One answer step. GPT-5.4 mini (low reasoning), the same instruction for every system.
- One judge. Claude Sonnet 5, three votes per answer; the majority counts, a three-way split takes the middle verdict.
- Success rate. Of the questions the answer model gets right when handed the right pages (the ceiling), the share each system gets right with its own search. The floor is the same model with no documents.
Audit of misses
Every answer that did not match the reference, for every system and for the ceiling, was classified by Claude Opus 5.5 as a real miss, an ambiguous question, a reference problem, or right but less complete. The audit is automated and did not see which system answered.
Prompts
Every prompt, verbatim. Placeholders in angle brackets are filled per question; nothing else was sent. Floor is the answer model with no documents; Ceiling is the same model handed the right pages.
File Search is the one system whose query the model writes. On FinanceBench and SECQUE, GPT-5.4 mini wrote it under these instructions, limited to one search; its answer was then discarded and replaced by the shared answer step.
INSTRUCTION
You answer questions using only the provided material. Answer directly and concisely, stating the fact asked for. If the material does not contain the answer, say so plainly instead of guessing.
MESSAGE
CONTEXT CHUNKS:
--- Source 1 ---
Document: <document title>
<passage text>
--- Source 2 ---
…
---
QUESTION: <the dataset's question, verbatim>SYSTEM PROMPT
You grade a retrieval assistant's answer against a reference answer.
Reply with ONLY a JSON object:
{"verdict": "correct" | "partially_correct" | "incorrect" | "refused",
"reference_suspect": true | false,
"reference_suspect_quote": "<verbatim quote from the ANSWER, or null>",
"matched_variant": <integer index, or null>,
"rationale": "<one sentence>"}
Definitions:
- "correct": conveys the same substance the question asked for.
- "partially_correct": part of that substance, with a material omission — or right
but hedged so heavily the reader could not act on it.
- "incorrect": CONTRADICTS the reference on something the question asked for.
- "refused": declines to answer, says the information is not available, or asks
for clarification instead of answering.
Scope rule — apply this first, it is the one graders get wrong:
The reference states what a correct answer must CONTAIN, never what it may not
also say. Anything the answer adds beyond the reference is OUTSIDE the
reference's scope: it can never lower the verdict. Extra findings, extra caveats,
additional true detail, more sources, richer formatting — all irrelevant to the
grade. Only a CONTRADICTION of something the question asked for counts against
the answer. "Mentions things the reference does not mention" is not a defect.
Numeric agreement — use these bands, do not invent a tolerance:
- within 10% of the reference value, or inside a stated confidence interval → treat as matching
- 10-25% off, same direction and same conclusion → at most "partially_correct"
- more than 25% off, or the opposite direction/conclusion → "incorrect"
- a DIFFERENT statistic that the answer clearly labels (per-trial where the
reference pooled, a subgroup where the reference used all trials) is not a
contradiction: grade "partially_correct" if the question asked for the
reference's aggregation level, "correct" if it did not.
matched_variant: when ALTERNATIVE ACCEPTABLE ANSWERS are supplied below, they are
every bit as correct as the primary reference - they record that this question has
more than one legitimate reading (a different trial, a different table, normalized
vs. raw figures, label vs. clinical study report). Grade against the primary
reference AND every alternative, and award the BEST verdict any of them yields.
Set matched_variant to the 0-based index of the alternative that produced it, or
null when the primary reference did. Never lower a verdict because the answer
matched an alternative rather than the primary - choosing between legitimate
sources is not the assistant's error.
reference_suspect: set true ONLY when the answer cites the corpus for something
that CONTRADICTS the reference, and put that verbatim quote from the answer in
reference_suspect_quote. No quote means no flag — leave it false and set the
quote to null.
Two failures this rule exists to prevent, both real:
- An answer that simply does not mention what the reference says is NOT evidence
the reference is wrong. It may be answering a narrower or different question.
A contract reference read "appointed as sole and exclusive agent"; the answer
said the party is called "the Customer". Both are true in the document, the
answer just addressed a different clause. That was flagged and should not have
been.
- Do not protect the reference because it is the reference. Where the answer
cites the document for a specific, checkable claim and the reference asserts
something the document does not support, say so — even though your verdict is
"incorrect". [One example omitted here: it came from a test that is not in this report.]
Still give your best verdict on the merits either way. This exists because gold
answers written from secondary material have repeatedly turned out to contradict
the source documents.
Grade only substance. Do not reward style, apologies or confidence. An answer
that is right for the wrong reason is still "correct" — judging the reasoning is
not your job.
MESSAGE
QUESTION:
<question>
REFERENCE ANSWER:
<the dataset's reference answer>
ASSISTANT'S ANSWER:
<the system's answer>INSTRUCTION
Answer as briefly as the question allows. State the specific answer first: the figure with its unit and the fiscal period it belongs to, or the yes/no with the one fact that decides it. When the answer has to be computed, give the inputs and the formula in one line. No preamble, no restating the question, no closing remarks. Keep citations.
SEARCH INSTRUCTIONS
- This is a document-grounded question set over company filings: ALWAYS search the documents before answering, including questions that read like general knowledge. The expected answer comes from the named company's filing for the named period, never from memory.
- When a question needs figures from more than one statement or period (an income statement AND a cash flow statement, two fiscal years), run a SEPARATE search for each figure instead of one combined query.
- Search again whenever the results are thin, come from another company or another period, or cover only part of the question. Refine the wording, or search for the missing figure specifically.
- Stop when further searching would add nothing — not merely when you have something to say.INSTRUCTION
You answer questions from your own knowledge. You have no documents and no search. Answer directly and concisely. If you do not know, say so in one sentence.
MESSAGE
QUESTION: <question>The answer step above, unchanged, with the benchmark's own evidence for the question as the context chunks.How to reproduce
Repeat the legal results
Download the datasets, set up each system as below, and grade with the prompts.
Datasets
- LegalBench-RAG (mini), ZeroEntropy, github.com/zeroentropy-ai/legalbenchrag, licence MIT
Used as published. This page reproduces no question, answer or passage from them.
How each system was set up
- OpenAI File Search. Vector store with automatic chunking; the file-search tool, 10 results, one search.
- OpenAI Direct Search. The same vector store, searched once with the question, 10 results.
- Azure AI Search. Import-wizard vectorization, 2,000-character chunks with 500 overlap, text-embedding-3-large; one hybrid query with the semantic ranker, top 10.
- Vertex AI Search. Standard edition, EU; layout parser, 500-token chunks with ancestor headings; one search, default ranking, query expansion off, top 10; quotation marks removed from the query (see below).
- Hippocampus. Default settings, nothing tuned per test; one search per question through the search API, search release of 18 October 2026 (LegalBench-RAG: the release of 19 October, no measurable difference).
How answers were graded
- Same documents. Every system indexed the identical files, pinned by checksum.
- One search per question. The question as written; only OpenAI File Search lets the model write its query.
- One answer step. GPT-5.4 mini (low reasoning), the same instruction for every system.
- One judge. Claude Sonnet 5, three votes per answer; the majority counts, a three-way split takes the middle verdict.
- Success rate. Of the questions the answer model gets right when handed the right pages (the ceiling), the share each system gets right with its own search. The floor is the same model with no documents.
Audit of misses
Every answer that did not match the reference, for every system and for the ceiling, was classified by Claude Opus 5.5 as a real miss, an ambiguous question, a reference problem, or right but less complete. The audit is automated and did not see which system answered.
Vertex AI Search and quotation marks
Vertex AI Search reads double quotation marks as an exact-phrase operator. With LegalBench-RAG’s quoted party names sent as written, 347 of its 373 quoted questions returned no passage, so for Vertex alone we removed the quotation marks from the search query. The answer step saw every question as written.
Prompts
Every prompt, verbatim. Placeholders in angle brackets are filled per question; nothing else was sent. Floor is the answer model with no documents; Ceiling is the same model handed the right pages.
INSTRUCTION
You answer questions using only the provided material. Answer directly and concisely, stating the fact asked for. If the material does not contain the answer, say so plainly instead of guessing.
MESSAGE
CONTEXT CHUNKS:
--- Source 1 ---
Document: <document title>
<passage text>
--- Source 2 ---
…
---
QUESTION: <the dataset's question, verbatim>SYSTEM PROMPT
You grade a retrieval assistant's answer against a reference answer.
Reply with ONLY a JSON object:
{"verdict": "correct" | "partially_correct" | "incorrect" | "refused",
"reference_suspect": true | false,
"reference_suspect_quote": "<verbatim quote from the ANSWER, or null>",
"matched_variant": <integer index, or null>,
"rationale": "<one sentence>"}
Definitions:
- "correct": conveys the same substance the question asked for.
- "partially_correct": part of that substance, with a material omission — or right
but hedged so heavily the reader could not act on it.
- "incorrect": CONTRADICTS the reference on something the question asked for.
- "refused": declines to answer, says the information is not available, or asks
for clarification instead of answering.
Scope rule — apply this first, it is the one graders get wrong:
The reference states what a correct answer must CONTAIN, never what it may not
also say. Anything the answer adds beyond the reference is OUTSIDE the
reference's scope: it can never lower the verdict. Extra findings, extra caveats,
additional true detail, more sources, richer formatting — all irrelevant to the
grade. Only a CONTRADICTION of something the question asked for counts against
the answer. "Mentions things the reference does not mention" is not a defect.
Numeric agreement — use these bands, do not invent a tolerance:
- within 10% of the reference value, or inside a stated confidence interval → treat as matching
- 10-25% off, same direction and same conclusion → at most "partially_correct"
- more than 25% off, or the opposite direction/conclusion → "incorrect"
- a DIFFERENT statistic that the answer clearly labels (per-trial where the
reference pooled, a subgroup where the reference used all trials) is not a
contradiction: grade "partially_correct" if the question asked for the
reference's aggregation level, "correct" if it did not.
matched_variant: when ALTERNATIVE ACCEPTABLE ANSWERS are supplied below, they are
every bit as correct as the primary reference - they record that this question has
more than one legitimate reading (a different trial, a different table, normalized
vs. raw figures, label vs. clinical study report). Grade against the primary
reference AND every alternative, and award the BEST verdict any of them yields.
Set matched_variant to the 0-based index of the alternative that produced it, or
null when the primary reference did. Never lower a verdict because the answer
matched an alternative rather than the primary - choosing between legitimate
sources is not the assistant's error.
reference_suspect: set true ONLY when the answer cites the corpus for something
that CONTRADICTS the reference, and put that verbatim quote from the answer in
reference_suspect_quote. No quote means no flag — leave it false and set the
quote to null.
Two failures this rule exists to prevent, both real:
- An answer that simply does not mention what the reference says is NOT evidence
the reference is wrong. It may be answering a narrower or different question.
A contract reference read "appointed as sole and exclusive agent"; the answer
said the party is called "the Customer". Both are true in the document, the
answer just addressed a different clause. That was flagged and should not have
been.
- Do not protect the reference because it is the reference. Where the answer
cites the document for a specific, checkable claim and the reference asserts
something the document does not support, say so — even though your verdict is
"incorrect". [One example omitted here: it came from a test that is not in this report.]
Still give your best verdict on the merits either way. This exists because gold
answers written from secondary material have repeatedly turned out to contradict
the source documents.
Grade only substance. Do not reward style, apologies or confidence. An answer
that is right for the wrong reason is still "correct" — judging the reasoning is
not your job.
MESSAGE
QUESTION:
<question>
REFERENCE ANSWER:
<the dataset's reference answer>
ASSISTANT'S ANSWER:
<the system's answer>INSTRUCTION
You answer questions from your own knowledge. You have no documents and no search. Answer directly and concisely. If you do not know, say so in one sentence.
MESSAGE
QUESTION: <question>The answer step above, unchanged, with the benchmark's own evidence for the question as the context chunks.How to reproduce
Repeat the healthcare results
The documents are public; the questions are not. This is what was run and how to check it.
Data
Ten public FDA documents, 2,198 pages: reviews, labels and a trial publication for Saxenda, Wegovy and Zepbound. Sixteen questions, each asked three times in independent conversations: 8 easy, 4 medium and 4 hard, one of them a control question the documents cannot answer, where declining is correct.
How each system was set up
- Hippocampus. Searches as often as it needs, up to 12 times, with hierarchical retrieval; it also gets an overview of the collection, a short search guide and answer instructions, refined in earlier runs on these questions.
- OpenAI File Search. All ten PDFs in one vector store with default chunking; up to 12 calls and 10 results per call, settings of this evaluation.
- OpenAI vector store search. The same store, one search with the question, 10 results.
- PageIndex. Documents indexed by PageIndex’s cloud service; in one step the model chose up to six sections from the ten document trees and answered from them, up to 12,000 characters each. This is our adaptation of PageIndex’s published tree search.
- No documents. The same model on its own.
GPT-5.4 mini wrote every answer: the one thing held fixed.
How answers were graded
GPT-5, three votes, majority, against the question set’s reference answers. Declining the control question counts as correct. Partially correct answers and declined answerable questions score zero. Before the runs, three reference answers were corrected against the source documents, and every system was graded against the same corrected set.
Repetition and the stronger model
OpenAI File Search ran the full set twice with identical settings and scored 38 and 34 of 48. The other systems ran once, so differences of a few answers can come from run-to-run variation alone. With Claude Opus 5 answering, the same Hippocampus configuration answered 93 of 96 across two full runs (47 and 46 of 48); the other systems were not run with it, so that figure is not a comparison.
Run it yourself
The ten documents are public: FDA publications and a peer-reviewed journal article. We can load them into a separate Hippocampus workspace, so you can put your own questions to it, and to the other systems with your own accounts. The questions and reference answers were written with a customer and belong to them; with that customer’s agreement, we can take qualified parties through them and every graded answer under NDA.
- 92% success rate on FinanceBench Of the questions a perfect search would solve, Hippocampus solves 92%. Next best: 77%.
- 86% of all FinanceBench questions answered right Every question counts, nothing filtered out. Next best: 70%.
- 85% of the FinanceBench evidence found Share of the passages needed to answer that the search found. Next best: 58%.
Head to head
FinanceBench: success rate by system
150 analyst questions on 84 US company filings, from Patronus AI. Each bar is the share of answerable questions the system gets right with its own search.
Read the full reportResult deep dive
What FinanceBench shows.
Figures, ratios and judgements about a company’s finances, on 84 annual, quarterly and current reports (10-K, 10-Q, 8-K). Close to real analyst work: one document per question, named in the question.
Head to head
SECQUE: success rate by system
565 expert questions on 45 SEC filings, from Ben Yoash et al. Each bar is the share of answerable questions the system gets right with its own search.
Read the full reportResult deep dive
What SECQUE shows.
Comparing and computing ratios, trends and risk on 45 SEC filings. Realistic analyst tasks: answers often need several passages and a calculation.
- 85.7% success rate Of the questions a perfect search would solve, Hippocampus solves 85.7%. Next best: 78.3%.
- 82% of all questions answered right Every question counts, nothing filtered out. Next best: 75%.
- 86% of the evidence found Share of the passages needed to answer that the search found. Next best: 75%.
Head to head
LegalBench-RAG: success rate by system
776 questions on 72 contracts and privacy policies, from ZeroEntropy. Each bar is the share of answerable questions the system gets right with its own search.
Read the full reportResult deep dive
What LegalBench-RAG shows.
NDAs, merger agreements, commercial contracts and privacy policies: 72 documents, the mini split of LegalBench-RAG. Close to legal review work: the answer is a passage in one named document among look-alike agreements.
- 100% retrieval accuracy Claude Opus 5 answering. A correct source document reached for every answerable question, every repetition.
- 97% correct Claude Opus 5 answering. Two full runs of the question set (47 of 48 and 46 of 48), including the control question declined in all six repetitions.
- 6/6 control question declined Claude Opus 5 answering. Asked something the documents cannot answer, it declined the control question in all six repetitions.
Head to head
Healthcare: answers right by system
Every system answered through GPT-5.4 mini, the same small model.
Read the full reportResult deep dive
What the FDA benchmark shows.
Every system with document access answered most of them (19 to 24 of 24), and the model alone answered half from its own knowledge, since these are public documents. Single-fact lookups separate the systems only a little.
See it on your own documents.
Free tier, no card, no API key. Import one document and ask it the question the alternatives get wrong.