Evaluate RAG abstention by labelling whether the authorised evidence supports an answer, then recording whether the assistant answers or declines. Report unsupported answers on unanswerable cases separately from unnecessary abstentions on answerable cases. Check factual correctness within the answers as a further step. A low hallucination count alone cannot tell you whether the assistant does useful work.

For an internal policy assistant, that distinction has a cost. An invented entitlement sends someone in the wrong direction. A bot that declines a documented policy question sends the same work back to a colleague. Both belong in the acceptance decision.

Define answerability against the evidence the assistant may use

For company-policy questions, “the model knows something similar” is an inadequate basis for answering. The relevant question is whether the current, applicable evidence available to this user supports the requested claim.

Consider a hypothetical consultancy asking its Teams assistant: “Can I expense a taxi home after a late client event?” A travel policy covering airport transfers does not settle that question. An answer quoting the airport rule can be accurate about that rule and still mislead the employee about the event.

Label the case before inspecting the assistant’s response. Record the applicable policy version and the clause that supports the expected answer. When no clause settles the question, specify the permitted outcome: explain the gap and route it to the policy owner. Abstention here means declining to assert the unsupported entitlement; the assistant can still give a useful next step.

Keep access failures distinct from evidence gaps. If a connector fails or a user cannot access a document, record that condition rather than claiming the policy does not exist. Do not expose restricted content in the explanation.

The structured-output guide addresses a related but earlier contract: whether the output can represent missing information. This evaluation checks whether the assistant makes the right decision when information is missing.

Use a worksheet that preserves both failure directions

The table below is a proposed evaluation design, not a measured Model Operator result or a vendor benchmark. Classify the substantive answer to the requested claim; a polite disclaimer followed by an unsupported answer still counts as answering.

Evidence labelAssistant answersAssistant abstains
Sufficient authorised evidenceCheck whether the answer is correct and applicableUnnecessary abstention
Insufficient authorised evidenceUnsupported answerAppropriate abstention; inspect the handoff

Use these denominators explicitly:

  • Unsupported-answer rate: cases answered despite insufficient evidence, divided by all insufficient-evidence cases.
  • For sufficient-evidence cases, divide unnecessary abstentions by the total in that group.
  • Answer coverage uses all evaluated cases as its denominator and counts cases receiving a substantive answer, regardless of correctness.
  • Accepted-answer yield counts correct, supported answers divided by all evaluated cases.

Publish the underlying counts alongside the rates. Mark a rate as not applicable if its denominator is zero. Coverage can increase because the system invents more answers; the unsupported-answer rate tells the reader whether that happened.

Also report answer correctness within the sufficient-evidence/answered cell. Choosing to answer is only the first decision. The assistant can retrieve the right document and misread its condition.

Change the evidence while holding the question constant

Build paired fixtures around actual questions the team expects to receive. For the taxi example, start with an approved clause that explicitly addresses late client events. The expected response should apply that clause and cite it.

Create a second fixture with that clause absent, leaving the airport-transfer material in place. The expected response should explain that the available policy does not settle the late-event question. This tests whether nearby but inapplicable evidence tempts the assistant into answering.

A conflicting-version fixture adds another decision: can the system establish which policy governs? If the fixture provides no authorised way to resolve the conflict, the assistant should leave the disputed entitlement open. A partial response must identify which part is supported and withhold the rest.

Keep complete-evidence cases in the suite whenever you tighten abstention instructions. Otherwise a change that suppresses every answer can appear successful. Retain the same fixture identifiers across versions so the team can see which previously useful answers disappeared.

Run the paired suite separately from a representative sample of normal enquiries. Deliberately removing evidence creates a challenge set; its failure rate is not an estimate of how frequently missing evidence occurs in production.

What the research supports, and what it leaves to your team

AbstentionBench, published in 2025, evaluates abstention across different kinds of unanswerable questions. Its findings challenge the assumption that a stronger reasoning model will automatically make better answer-or-abstain decisions. They describe the models and tasks tested in that study, not a ranking of today’s assistants or a forecast for your deployment.

Zhou and colleagues examine refusal in retrieval-augmented models, including over-refusal under irrelevant retrieval. They report that improved refusal behaviour does not necessarily imply better calibration or higher overall accuracy. Their experiments support testing these properties separately. They do not establish an acceptance threshold for a company-policy bot.

The DeepEval abstention classifier provides a practical starting point for context-conditioned grading. Its documentation asks evaluators to supply retrieval context and includes labels for abstaining when evidence is absent, answering when it is present, and fabricating when it is absent.

The worksheet above explicitly adds the business-relevant case of sufficient evidence followed by abstention. Do not assume an automated judge’s default labels account for every cell. Review ambiguous cases against the policy owner’s labels, especially where a response mixes a correct partial answer with an unsupported conclusion.

Set the acceptance decision around the remaining workload

Agree thresholds by consequence before comparing prompt or retrieval changes. A general office question and a contractual entitlement do not warrant identical tolerance for unsupported answers. Keep the more consequential cases visible rather than averaging them into routine enquiries.

For abstentions, record where the question goes and whether anyone resolves it. A message saying “ask your manager” transfers the task; a tracked handoff allows the team to measure the work that remains. Include reviewer time when judging whether the assistant reduces the operating burden.

Use the broader pilot success-criteria guide to place this worksheet alongside adoption and commercial acceptance. The decision to expand should require evidence that the assistant preserves useful supported answers while containing unsupported ones, with a workable route for what it cannot settle.

Model Operator is Alexander Chrisostomou’s Dubai-based AI product studio. For a Company Brain + Slack / Teams Bots engagement, bring one policy workflow and a sample of its unresolved questions to a Model Operator build conversation. Define the answer boundary and the handoff owner before agreeing the implementation scope.