When your AI assistant learns from itself

Most AI assistants for business now remember things: your preferences, your clients, how you like work done.
The question almost nobody asks a supplier is who wrote each memory.
If the assistant can save its own conclusions as facts, its mistakes come back next week looking exactly like something you told it.

Memory is a set of notes the assistant writes

When an assistant “remembers”, it saves a short note and reads it back in later conversations.
Some notes come from you: “we invoice on the 25th”.
Others it writes itself, from what it inferred: “this client prefers phone calls”.
A week later, both look the same to it.

A new starter’s notes work the same way.
They help, until a wrong guess in week one turns into office lore by week six and nobody can say where it came from.

What happens when a system feeds on its own output

A team led by Oxford researchers trained language models on text produced by earlier versions of the same models, generation after generation.
Their 2024 paper in Nature calls the result model collapse.
The rare cases disappear first, and the damage does not reverse.

Your assistant’s memory is a much smaller loop, and the same thing happens.
It loses the unusual cases first: the odd client, the exception to your policy, the one supplier who works differently.
Those are usually what you needed it to know.

OpenAI saw a version of this in April 2025.
It rolled back a ChatGPT update that had become too agreeable, and its write-up named a new reward signal built from users’ thumbs-up and thumbs-down as a likely cause.
It added that memory made the effect worse in some cases.

Other people can write to its memory too

If an assistant saves notes from what it reads, then whoever wrote what it reads can write to its memory.
In 2024 the security researcher Johann Rehberger showed that a web page or document could plant instructions in ChatGPT’s long-term memory, where they stayed for later conversations.
OpenAI closed the route he used.

In 2025 researchers published an attack called MINJA that wrote false records into an AI agent’s memory through ordinary questions alone.
Over 95% of their injections succeeded in testing.
A 2026 follow-up found that real memories already in place made the attack much less effective.
A new assistant with little history is the most exposed.

The rule we use: only people teach

We build an AI agent, so we had to decide this.
Our rule is that the agent’s own conclusions never become rules.
Only something a person wrote, an instruction or a correction, can do that.

• Its own notes carry a label. Anything the agent inferred says so, and stays on the dashboard where you can confirm or hide it.
• They count for less. When it looks something up, its own inferences weigh 30% less than what you told it.
  They still come back, so a useful one still helps.
• They never prove a rule. A rule needs a person behind it.
  When the system cannot tell who wrote a note, it treats the note as not from a person.

Those two defaults run in opposite directions on purpose.
Demoting a good note by mistake costs one slightly worse answer.
Teaching the assistant something false costs every answer after it, and nobody can trace it.

If a supplier cannot answer the second question, nobody can tell you where any belief came from.
Our AI agent works to the rule above, and our trust page sets out the other checks it runs before it answers.