Benchmarks ·
Hippocampus on FDA GLP-1 documents: 97% correct with Claude Opus 5
Over 2,198 pages of public FDA documents, Hippocampus got 93 of 96 answers right (97%) with Claude Opus 5 answering, and invented nothing. With one small model writing every answer for five systems, Hippocampus answered 40 of 48, the most in the comparison.
On this pageThe test
The test
Ten public documents on three GLP-1 therapies for weight management: Saxenda and Wegovy from Novo Nordisk, Zepbound from Eli Lilly. FDA clinical and multidisciplinary reviews of 230 to 390 pages, prescribing information, approval packages, and the STEP 1 trial in the New England Journal of Medicine. The same measure appears in several places, for different trials and time points, so a correct answer depends on the right passage, not just the right document.
Sixteen expert questions, written with a customer: 8 easy (single facts), 4 medium (comparisons across documents) and 4 hard (synthesis across the therapies, plus one control question the documents cannot answer, where declining is correct). Each question was asked three times, in independent conversations.
We report two runs:
- The comparison (): five systems, one small answering model, GPT-5.4 mini, for all of them.
- The frontier run (16 to 17 September 2026): Hippocampus with Claude Opus 5 answering, the full set twice, plus two low-cost models.
How to read the numbers
- Correct: a separate grading model compared each answer with the reference answer, three votes, majority verdict. Declining the control question counts as correct.
- Weighted: easy questions count once, medium twice, hard three times.
- Right every time: of the 16 questions, those answered correctly in every repetition; for Claude Opus 5, in all six attempts across both runs.
- Source found: retrieval reached at least one document that holds the answer.
- Cost: per answer.
The comparison: five systems, one small model
| System | Correct of 48 | Weighted | Easy (24) | Medium (12) | Hard (12) |
|---|---|---|---|---|---|
| Hippocampus | 40 | 79% | 23 | 8 | 9 |
| PageIndex | 33 | 65% | 19 | 6 | 8 |
| OpenAI File Search, average of 2 runs | 36 | 64% | 24 | 6.5 | 5.5 |
| OpenAI vector store search | 29 | 46% | 21 | 6 | 2 |
| No documents | 15 | 25% | 12 | 0 | 3 |
OpenAI File Search ran twice with identical settings and scored 38 and 34, so the gap between Hippocampus and File Search is within run-to-run variation. The gaps to PageIndex, the vector store and the model alone are larger.
Lookups are easy for everyone. The difference is on comparisons and synthesis: of the 24 medium and hard answers, Hippocampus got 17 right, including the control question, PageIndex 14, File Search 12 on average (14 and 10) and the vector store 8.
On the control question, PageIndex declined every time. Hippocampus said each time that the documents cannot support a ranking, and in two of three answers offered the approval order as a labelled proxy. File Search gave a ranking in 4 of 6 attempts, the vector store in 1 of 3.
Correct answers by difficulty, one small model for every system
Each bar counts correct answers out of 24: on the left the 8 easy questions, on the right the 4 medium and 4 hard questions, each asked three times.
Easy questions, of 24
- Hippocampus 23 of 24
- PageIndex 19 of 24
- OpenAI File Search, average of 2 runs 24 of 24
- OpenAI vector store search 21 of 24
- No documents 12 of 24
Medium and hard questions, of 24
- Hippocampus 17 of 24
- PageIndex 14 of 24
- OpenAI File Search, average of 2 runs 12 of 24
- OpenAI vector store search 8 of 24
- No documents 3 of 24
Table view: every count behind the chart
| Group | Name | Value | Note |
|---|---|---|---|
| Easy questions, of 24 | Hippocampus | 23 of 24 | |
| Easy questions, of 24 | PageIndex | 19 of 24 | Our adaptation of the published tree-search approach |
| Easy questions, of 24 | OpenAI File Search, average of 2 runs | 24 of 24 | Average of two identical runs |
| Easy questions, of 24 | OpenAI vector store search | 21 of 24 | |
| Easy questions, of 24 | No documents | 12 of 24 | The model's own knowledge |
| Medium and hard questions, of 24 | Hippocampus | 17 of 24 | Includes the control question |
| Medium and hard questions, of 24 | PageIndex | 14 of 24 | |
| Medium and hard questions, of 24 | OpenAI File Search, average of 2 runs | 12 of 24 | Average of two runs: 14 and 10 |
| Medium and hard questions, of 24 | OpenAI vector store search | 8 of 24 | |
| Medium and hard questions, of 24 | No documents | 3 of 24 |
The frontier run: same retrieval, three answering models
| Answering model | Correct | Right every time | Invented | Source found | Cost |
|---|---|---|---|---|---|
| Claude Opus 5, two runs | 93 of 96 | 15 of 16 | 0 | 100% | $0.61 |
| GPT-5.4 mini | 40 of 48 | 11 of 16 | 0 | 98% | $0.12 |
| Claude Haiku 4.5 | 38 of 48 | 11 of 16 | 0 | 93% | $0.09 |
Claude Opus 5 scored 47 and 46 of 48 in two runs a few hours apart, and declined the control question 6 of 6 times. The one-answer spread is the honest measure of repeatability. The run was set up for accuracy, not economy: extended reasoning and as many searches as each question needed. Low-cost models on the same retrieval answer 38 to 40 of 48 at $0.09 to $0.12 per answer, never inventing an answer.
Questions right every time, three answering models on Hippocampus
Each bar counts the questions, of 16, answered correctly in every repetition, with the same Hippocampus retrieval and a different answering model.
- Claude Opus 5, two runs 15 of 16
- GPT-5.4 mini 11 of 16
- Claude Haiku 4.5 11 of 16
Table view: every count behind the chart
| Name | Value | Note |
|---|---|---|
| Claude Opus 5, two runs | 15 of 16 | All six attempts; 93 of 96 answers correct |
| GPT-5.4 mini | 11 of 16 | Three attempts; 40 of 48 answers correct |
| Claude Haiku 4.5 | 11 of 16 | Three attempts; 38 of 48 answers correct |
The miss
All three Opus misses came from one question: which safety signals all three therapies share. A heart-rate increase is reported for all three, but the FDA lists it under Warnings and Precautions for Saxenda and Wegovy and under Adverse Reactions for Zepbound. An answer that compares only the warnings lists concludes, from a true premise, that the signal is not shared. Asked from memory, two of three frontier models got the placement wrong, in opposite directions.
Hippocampus now guides the model to search each product's own documents before stating that something is absent. With that, it counted the signal as shared in 3 of 6 repetitions; the previous release did so in 1 of 6.
Honest notes
- Hippocampus configurations tested with GPT-5.4 mini on 16 to 17 September scored between 37 and 40 of 48. The comparison shows a run that scored 40, made earlier the same day on the deployed service, separately from the other systems.
- The search guide Hippocampus gives the model was refined in earlier runs on these same sixteen questions. The other systems ran on vendor defaults plus a one-sentence instruction to answer only from the material.
- The comparison's grading model is GPT-5. The answering model and two of the systems tested are also OpenAI's.
- Before the comparison, three reference answers were corrected against the source documents, and every system was graded against the same corrected set.
- PageIndex ran as our adaptation of its published tree-search approach: one step, up to six sections, 12,000 characters per section. PageIndex's own retrieval and chat services were not used.
- About 410 of the 2,198 pages have no text layer, including all 300 pages of a 2009 pharmacology review. OpenAI's vector store held no text from that review; Hippocampus and PageIndex read scanned pages. No system cited it, and the one question that draws on it can also be answered from a label.
- Three repetitions separate stable behaviour from chance. They do not give a confidence interval.
Speed
These are times for a whole answer, including every search the model chose to run. In the comparison, Hippocampus took a median of 20 seconds, against 5 to 8 seconds for the other systems with documents, and over two minutes for its slowest 5%. Answers that needed one search took about 10 seconds; the slow ones were comparisons and synthesis, where the model searched many times. With Claude Opus 5 the median was 46 seconds.
How we tested
Every system worked from the same ten PDFs and the same sixteen questions. In Hippocampus, the model searches the collection as often as it needs, using hierarchical retrieval (patent pending), and cites document, section and page for every claim. The full method is on the evidence page under the Healthcare tab.
The documents are public. The questions and reference answers belong to the customer they were written with; with their agreement, we take qualified parties through every graded answer under NDA.


