Benchmarks ·

Hippocampus on FDA GLP-1 documents: 97% correct with Claude Opus 5

Over 2,198 pages of public FDA documents, Hippocampus got 93 of 96 answers right (97%) with Claude Opus 5 answering, and invented nothing. With one small model writing every answer for five systems, Hippocampus answered 40 of 48, the most in the comparison.

On this pageThe test
  1. The test
  2. How to read the numbers
  3. The comparison: five systems, one small model
  4. The frontier run: same retrieval, three answering models
  5. The miss
  6. Honest notes
  7. Speed
  8. How we tested

The test

Ten public documents on three GLP-1 therapies for weight management: Saxenda and Wegovy from Novo Nordisk, Zepbound from Eli Lilly. FDA clinical and multidisciplinary reviews of 230 to 390 pages, prescribing information, approval packages, and the STEP 1 trial in the New England Journal of Medicine. The same measure appears in several places, for different trials and time points, so a correct answer depends on the right passage, not just the right document.

Sixteen expert questions, written with a customer: 8 easy (single facts), 4 medium (comparisons across documents) and 4 hard (synthesis across the therapies, plus one control question the documents cannot answer, where declining is correct). Each question was asked three times, in independent conversations.

We report two runs:

  • The comparison (): five systems, one small answering model, GPT-5.4 mini, for all of them.
  • The frontier run (16 to 17 September 2026): Hippocampus with Claude Opus 5 answering, the full set twice, plus two low-cost models.

How to read the numbers

  • Correct: a separate grading model compared each answer with the reference answer, three votes, majority verdict. Declining the control question counts as correct.
  • Weighted: easy questions count once, medium twice, hard three times.
  • Right every time: of the 16 questions, those answered correctly in every repetition; for Claude Opus 5, in all six attempts across both runs.
  • Source found: retrieval reached at least one document that holds the answer.
  • Cost: per answer.

The comparison: five systems, one small model

System Correct of 48 Weighted Easy (24) Medium (12) Hard (12)
Hippocampus 40 79% 23 8 9
PageIndex 33 65% 19 6 8
OpenAI File Search, average of 2 runs 36 64% 24 6.5 5.5
OpenAI vector store search 29 46% 21 6 2
No documents 15 25% 12 0 3

OpenAI File Search ran twice with identical settings and scored 38 and 34, so the gap between Hippocampus and File Search is within run-to-run variation. The gaps to PageIndex, the vector store and the model alone are larger.

Lookups are easy for everyone. The difference is on comparisons and synthesis: of the 24 medium and hard answers, Hippocampus got 17 right, including the control question, PageIndex 14, File Search 12 on average (14 and 10) and the vector store 8.

On the control question, PageIndex declined every time. Hippocampus said each time that the documents cannot support a ranking, and in two of three answers offered the approval order as a labelled proxy. File Search gave a ranking in 4 of 6 attempts, the vector store in 1 of 3.

Correct answers by difficulty, one small model for every system

Each bar counts correct answers out of 24: on the left the 8 easy questions, on the right the 4 medium and 4 hard questions, each asked three times.

Easy questions, of 24

  • Hippocampus 23 of 24
  • PageIndex 19 of 24
  • OpenAI File Search, average of 2 runs 24 of 24
  • OpenAI vector store search 21 of 24
  • No documents 12 of 24

Medium and hard questions, of 24

  • Hippocampus 17 of 24
  • PageIndex 14 of 24
  • OpenAI File Search, average of 2 runs 12 of 24
  • OpenAI vector store search 8 of 24
  • No documents 3 of 24
Table view: every count behind the chart
Correct answers by difficulty, one small model for every system
GroupNameValueNote
Easy questions, of 24Hippocampus23 of 24
Easy questions, of 24PageIndex19 of 24Our adaptation of the published tree-search approach
Easy questions, of 24OpenAI File Search, average of 2 runs24 of 24Average of two identical runs
Easy questions, of 24OpenAI vector store search21 of 24
Easy questions, of 24No documents12 of 24The model's own knowledge
Medium and hard questions, of 24Hippocampus17 of 24Includes the control question
Medium and hard questions, of 24PageIndex14 of 24
Medium and hard questions, of 24OpenAI File Search, average of 2 runs12 of 24Average of two runs: 14 and 10
Medium and hard questions, of 24OpenAI vector store search8 of 24
Medium and hard questions, of 24No documents3 of 24

n = 16 questions, three attempts each, 48 answers per system · answering model GPT-5.4 mini for every system · grader GPT-5, three votes, majority verdict, against a reference set with three answers corrected beforehand · 17 September 2026 · Thalius FDA comparison report. OpenAI File Search ran twice and shows the average; the other systems ran once. Hippocampus configurations scored 37 to 40 of 48; this run scored 40.

The frontier run: same retrieval, three answering models

Answering model Correct Right every time Invented Source found Cost
Claude Opus 5, two runs 93 of 96 15 of 16 0 100% $0.61
GPT-5.4 mini 40 of 48 11 of 16 0 98% $0.12
Claude Haiku 4.5 38 of 48 11 of 16 0 93% $0.09

Claude Opus 5 scored 47 and 46 of 48 in two runs a few hours apart, and declined the control question 6 of 6 times. The one-answer spread is the honest measure of repeatability. The run was set up for accuracy, not economy: extended reasoning and as many searches as each question needed. Low-cost models on the same retrieval answer 38 to 40 of 48 at $0.09 to $0.12 per answer, never inventing an answer.

Questions right every time, three answering models on Hippocampus

Each bar counts the questions, of 16, answered correctly in every repetition, with the same Hippocampus retrieval and a different answering model.

  • Claude Opus 5, two runs 15 of 16
  • GPT-5.4 mini 11 of 16
  • Claude Haiku 4.5 11 of 16
Table view: every count behind the chart
Questions right every time, three answering models on Hippocampus
NameValueNote
Claude Opus 5, two runs15 of 16All six attempts; 93 of 96 answers correct
GPT-5.4 mini11 of 16Three attempts; 40 of 48 answers correct
Claude Haiku 4.511 of 16Three attempts; 38 of 48 answers correct

n = 16 questions · Claude Opus 5: three attempts in each of two runs, six per question; GPT-5.4 mini and Claude Haiku 4.5: three attempts per question · a separate grading model, three votes, majority verdict · 16 to 17 September 2026 · Thalius FDA evaluation.

The miss

All three Opus misses came from one question: which safety signals all three therapies share. A heart-rate increase is reported for all three, but the FDA lists it under Warnings and Precautions for Saxenda and Wegovy and under Adverse Reactions for Zepbound. An answer that compares only the warnings lists concludes, from a true premise, that the signal is not shared. Asked from memory, two of three frontier models got the placement wrong, in opposite directions.

Hippocampus now guides the model to search each product's own documents before stating that something is absent. With that, it counted the signal as shared in 3 of 6 repetitions; the previous release did so in 1 of 6.

Honest notes

  • Hippocampus configurations tested with GPT-5.4 mini on 16 to 17 September scored between 37 and 40 of 48. The comparison shows a run that scored 40, made earlier the same day on the deployed service, separately from the other systems.
  • The search guide Hippocampus gives the model was refined in earlier runs on these same sixteen questions. The other systems ran on vendor defaults plus a one-sentence instruction to answer only from the material.
  • The comparison's grading model is GPT-5. The answering model and two of the systems tested are also OpenAI's.
  • Before the comparison, three reference answers were corrected against the source documents, and every system was graded against the same corrected set.
  • PageIndex ran as our adaptation of its published tree-search approach: one step, up to six sections, 12,000 characters per section. PageIndex's own retrieval and chat services were not used.
  • About 410 of the 2,198 pages have no text layer, including all 300 pages of a 2009 pharmacology review. OpenAI's vector store held no text from that review; Hippocampus and PageIndex read scanned pages. No system cited it, and the one question that draws on it can also be answered from a label.
  • Three repetitions separate stable behaviour from chance. They do not give a confidence interval.

Speed

These are times for a whole answer, including every search the model chose to run. In the comparison, Hippocampus took a median of 20 seconds, against 5 to 8 seconds for the other systems with documents, and over two minutes for its slowest 5%. Answers that needed one search took about 10 seconds; the slow ones were comparisons and synthesis, where the model searched many times. With Claude Opus 5 the median was 46 seconds.

How we tested

Every system worked from the same ten PDFs and the same sixteen questions. In Hippocampus, the model searches the collection as often as it needs, using hierarchical retrieval (patent pending), and cites document, section and page for every claim. The full method is on the evidence page under the Healthcare tab.

The documents are public. The questions and reference answers belong to the customer they were written with; with their agreement, we take qualified parties through every graded answer under NDA.

By Andreas Dahlström, Head of Deeptech Team, Co-Founder

  • 2026
  • Benchmarks