Child of #12508. Covers its third request: a self-check for high-specificity facts before the answer is presented.
The failure this addresses
In the session #12508 was filed from, web_search returned the correct bus number verbatim ("Bus 22 and Minibus 322") and the delivered answer named a different one. Ranking and citing sources (#17506) makes the right result easier to reach; it does not stop the model from emitting a number that appears in none of them.
A route number, an address, a platform, a time, a price and a phone number share a property that prose does not: each is a short token that either occurs in the retrieved text or does not. That makes the check mechanical rather than a matter of judgement, which is why it is worth doing at all.
Acceptance criteria
Notes
The fourth and sixth criteria are the substance. A grounding check that cannot distinguish nothing to check from checked and clean will report clean on the day its extractor breaks, and the answer it blesses will be exactly as wrong as the one that prompted #12508. Same shape as the instances catalogued on #15826.
Deliberately not specified here: whether an ungrounded span blocks the answer, annotates it, or triggers a re-search. That is a product decision about failure behaviour, and it should be decided with evidence from the first three criteria rather than guessed now.
Depends on #17506 for the per-entry source text this checks against.
Child of #12508. Covers its third request: a self-check for high-specificity facts before the answer is presented.
The failure this addresses
In the session #12508 was filed from,
web_searchreturned the correct bus number verbatim ("Bus 22 and Minibus 322") and the delivered answer named a different one. Ranking and citing sources (#17506) makes the right result easier to reach; it does not stop the model from emitting a number that appears in none of them.A route number, an address, a platform, a time, a price and a phone number share a property that prose does not: each is a short token that either occurs in the retrieved text or does not. That makes the check mechanical rather than a matter of judgement, which is why it is worth doing at all.
Acceptance criteria
Notes
The fourth and sixth criteria are the substance. A grounding check that cannot distinguish nothing to check from checked and clean will report clean on the day its extractor breaks, and the answer it blesses will be exactly as wrong as the one that prompted #12508. Same shape as the instances catalogued on #15826.
Deliberately not specified here: whether an ungrounded span blocks the answer, annotates it, or triggers a re-search. That is a product decision about failure behaviour, and it should be decided with evidence from the first three criteria rather than guessed now.
Depends on #17506 for the per-entry source text this checks against.