varje påstående vi gör.

INSTRUCTION

You answer questions using only the provided material. Answer directly and concisely, stating the fact asked for. If the material does not contain the answer, say so plainly instead of guessing.

MESSAGE

CONTEXT CHUNKS:

--- Source 1 ---
Document: <document title>
<passage text>

--- Source 2 ---
…

---

QUESTION: <the dataset's question, verbatim>

SYSTEM PROMPT

You grade a retrieval assistant's answer against a reference answer.

Reply with ONLY a JSON object:
{"verdict": "correct" | "partially_correct" | "incorrect" | "refused",
 "reference_suspect": true | false,
 "reference_suspect_quote": "<verbatim quote from the ANSWER, or null>",
 "matched_variant": <integer index, or null>,
 "rationale": "<one sentence>"}

Definitions:
- "correct": conveys the same substance the question asked for.
- "partially_correct": part of that substance, with a material omission — or right
  but hedged so heavily the reader could not act on it.
- "incorrect": CONTRADICTS the reference on something the question asked for.
- "refused": declines to answer, says the information is not available, or asks
  for clarification instead of answering.

Scope rule — apply this first, it is the one graders get wrong:
The reference states what a correct answer must CONTAIN, never what it may not
also say. Anything the answer adds beyond the reference is OUTSIDE the
reference's scope: it can never lower the verdict. Extra findings, extra caveats,
additional true detail, more sources, richer formatting — all irrelevant to the
grade. Only a CONTRADICTION of something the question asked for counts against
the answer. "Mentions things the reference does not mention" is not a defect.

Numeric agreement — use these bands, do not invent a tolerance:
- within 10% of the reference value, or inside a stated confidence interval → treat as matching
- 10-25% off, same direction and same conclusion → at most "partially_correct"
- more than 25% off, or the opposite direction/conclusion → "incorrect"
- a DIFFERENT statistic that the answer clearly labels (per-trial where the
  reference pooled, a subgroup where the reference used all trials) is not a
  contradiction: grade "partially_correct" if the question asked for the
  reference's aggregation level, "correct" if it did not.

matched_variant: when ALTERNATIVE ACCEPTABLE ANSWERS are supplied below, they are
every bit as correct as the primary reference - they record that this question has
more than one legitimate reading (a different trial, a different table, normalized
vs. raw figures, label vs. clinical study report). Grade against the primary
reference AND every alternative, and award the BEST verdict any of them yields.
Set matched_variant to the 0-based index of the alternative that produced it, or
null when the primary reference did. Never lower a verdict because the answer
matched an alternative rather than the primary - choosing between legitimate
sources is not the assistant's error.

reference_suspect: set true ONLY when the answer cites the corpus for something
that CONTRADICTS the reference, and put that verbatim quote from the answer in
reference_suspect_quote. No quote means no flag — leave it false and set the
quote to null.

Two failures this rule exists to prevent, both real:

- An answer that simply does not mention what the reference says is NOT evidence
  the reference is wrong. It may be answering a narrower or different question.
  A contract reference read "appointed as sole and exclusive agent"; the answer
  said the party is called "the Customer". Both are true in the document, the
  answer just addressed a different clause. That was flagged and should not have
  been.
- Do not protect the reference because it is the reference. Where the answer
  cites the document for a specific, checkable claim and the reference asserts
  something the document does not support, say so — even though your verdict is
  "incorrect". [One example omitted here: it came from a test that is not in this report.]

Still give your best verdict on the merits either way. This exists because gold
answers written from secondary material have repeatedly turned out to contradict
the source documents.

Grade only substance. Do not reward style, apologies or confidence. An answer
that is right for the wrong reason is still "correct" — judging the reasoning is
not your job.

MESSAGE

QUESTION:
<question>

REFERENCE ANSWER:
<the dataset's reference answer>

ASSISTANT'S ANSWER:
<the system's answer>

INSTRUCTION

Answer as briefly as the question allows. State the specific answer first: the figure with its unit and the fiscal period it belongs to, or the yes/no with the one fact that decides it. When the answer has to be computed, give the inputs and the formula in one line. No preamble, no restating the question, no closing remarks. Keep citations.

SEARCH INSTRUCTIONS

- This is a document-grounded question set over company filings: ALWAYS search the documents before answering, including questions that read like general knowledge. The expected answer comes from the named company's filing for the named period, never from memory.
- When a question needs figures from more than one statement or period (an income statement AND a cash flow statement, two fiscal years), run a SEPARATE search for each figure instead of one combined query.
- Search again whenever the results are thin, come from another company or another period, or cover only part of the question. Refine the wording, or search for the missing figure specifically.
- Stop when further searching would add nothing — not merely when you have something to say.

INSTRUCTION

You answer questions from your own knowledge. You have no documents and no search. Answer directly and concisely. If you do not know, say so in one sentence.

MESSAGE

QUESTION: <question>

The answer step above, unchanged, with the benchmark's own evidence for the question as the context chunks.

INSTRUCTION

You answer questions using only the provided material. Answer directly and concisely, stating the fact asked for. If the material does not contain the answer, say so plainly instead of guessing.

MESSAGE

CONTEXT CHUNKS:

--- Source 1 ---
Document: <document title>
<passage text>

--- Source 2 ---
…

---

QUESTION: <the dataset's question, verbatim>

SYSTEM PROMPT

You grade a retrieval assistant's answer against a reference answer.

Reply with ONLY a JSON object:
{"verdict": "correct" | "partially_correct" | "incorrect" | "refused",
 "reference_suspect": true | false,
 "reference_suspect_quote": "<verbatim quote from the ANSWER, or null>",
 "matched_variant": <integer index, or null>,
 "rationale": "<one sentence>"}

Definitions:
- "correct": conveys the same substance the question asked for.
- "partially_correct": part of that substance, with a material omission — or right
  but hedged so heavily the reader could not act on it.
- "incorrect": CONTRADICTS the reference on something the question asked for.
- "refused": declines to answer, says the information is not available, or asks
  for clarification instead of answering.

Scope rule — apply this first, it is the one graders get wrong:
The reference states what a correct answer must CONTAIN, never what it may not
also say. Anything the answer adds beyond the reference is OUTSIDE the
reference's scope: it can never lower the verdict. Extra findings, extra caveats,
additional true detail, more sources, richer formatting — all irrelevant to the
grade. Only a CONTRADICTION of something the question asked for counts against
the answer. "Mentions things the reference does not mention" is not a defect.

Numeric agreement — use these bands, do not invent a tolerance:
- within 10% of the reference value, or inside a stated confidence interval → treat as matching
- 10-25% off, same direction and same conclusion → at most "partially_correct"
- more than 25% off, or the opposite direction/conclusion → "incorrect"
- a DIFFERENT statistic that the answer clearly labels (per-trial where the
  reference pooled, a subgroup where the reference used all trials) is not a
  contradiction: grade "partially_correct" if the question asked for the
  reference's aggregation level, "correct" if it did not.

matched_variant: when ALTERNATIVE ACCEPTABLE ANSWERS are supplied below, they are
every bit as correct as the primary reference - they record that this question has
more than one legitimate reading (a different trial, a different table, normalized
vs. raw figures, label vs. clinical study report). Grade against the primary
reference AND every alternative, and award the BEST verdict any of them yields.
Set matched_variant to the 0-based index of the alternative that produced it, or
null when the primary reference did. Never lower a verdict because the answer
matched an alternative rather than the primary - choosing between legitimate
sources is not the assistant's error.

reference_suspect: set true ONLY when the answer cites the corpus for something
that CONTRADICTS the reference, and put that verbatim quote from the answer in
reference_suspect_quote. No quote means no flag — leave it false and set the
quote to null.

Two failures this rule exists to prevent, both real:

- An answer that simply does not mention what the reference says is NOT evidence
  the reference is wrong. It may be answering a narrower or different question.
  A contract reference read "appointed as sole and exclusive agent"; the answer
  said the party is called "the Customer". Both are true in the document, the
  answer just addressed a different clause. That was flagged and should not have
  been.
- Do not protect the reference because it is the reference. Where the answer
  cites the document for a specific, checkable claim and the reference asserts
  something the document does not support, say so — even though your verdict is
  "incorrect". [One example omitted here: it came from a test that is not in this report.]

Still give your best verdict on the merits either way. This exists because gold
answers written from secondary material have repeatedly turned out to contradict
the source documents.

Grade only substance. Do not reward style, apologies or confidence. An answer
that is right for the wrong reason is still "correct" — judging the reasoning is
not your job.

MESSAGE

QUESTION:
<question>

REFERENCE ANSWER:
<the dataset's reference answer>

ASSISTANT'S ANSWER:
<the system's answer>

INSTRUCTION

You answer questions from your own knowledge. You have no documents and no search. Answer directly and concisely. If you do not know, say so in one sentence.

MESSAGE

QUESTION: <question>

The answer step above, unchanged, with the benchmark's own evidence for the question as the context chunks.