Checking an AI agent's answer against its sources

An AI agent works by looking things up and then writing an answer.
Most of the time the answer matches what it found.
Sometimes it adds a web address nobody gave it, a reference number that looks right, or a count of records that never existed.
We check every answer for those before anyone sees it.

What an agent invents

An agent does the most harm when what it invents looks like data.
An invented opinion is easy to spot.
A made-up invoice number, link or customer ID looks exactly like a real one, and someone may act on it.

These have a recognisable shape, so you can check them without asking another AI.
The agent kept a record of everything its tools returned.
If a reference number in the answer appears nowhere in that record, the agent did not read it anywhere.

The five checks

• Invented references. Every link, ID and code in the answer must appear in something a tool returned.
  The check allows links to well-known public documentation.
• Sources that do not say it. When the agent cites a source, the details in that sentence must be in that source.
  A real citation attached to the wrong claim is caught.
• Counts. The check compares “your table has 6 rows” with the counts the tools returned.
• Contradictions. In a task with several steps, the check flags the same thing given two different versions or values.
• What it saves, as well as what it says. A reference number the agent saves into a table must appear in what it read.

When a check fails, the agent gets one chance to rewrite, with the evidence in front of it.
We never edit the answer ourselves, because then nobody could see what the agent claimed.

The time our own check was wrong

A checker that fires on a real answer does damage too.
On 23 October our agent produced a list of firms with a link to each website.
The check flagged several real links as invented, and the agent dropped those firms from its answer.

The cause was punctuation.
A link written inside brackets ended in a closing bracket, and the check read that bracket as part of the address.
The address with a bracket on the end was not in the evidence, so it looked invented.

We fixed it the next day.
When we later moved the checks into a separate package, we replayed every input the old code had seen in our tests through both versions.
The replay found that the fix had been left out of the copy, so it went into the package before the platform used it.

What the checks cannot do

They check shapes, not meaning.
A wrong date or a wrong summary of a document passes, because dates and plain words match the evidence by chance too often to judge.
Two checks, counts and written values, only warn by default for the same reason.

So the checks are one layer.
The others are keeping the tool results the agent used, showing you where each claim came from, and asking a person before anything is sent or changed.

We have published the checks free, as @opafex/grounding on npm, with the source on GitHub.
They run without any AI model.
Our AI agent runs them on every answer, and our trust page sets out the other checks.