Nothing here remembers.

The part almost nobody knows

Every time you send a message to a chatbot, the whole conversation is sent back to it and read again from the beginning. It is not remembering the last thing it told you. There is no thread of experience running through the exchange, no continuity sitting behind the answers. Each turn, it reads a record and works out what it must have been thinking.

That is a strange kind of mind to have. Its entire past is a document, and the document arrives from outside. Which raises the question this site exists to answer:

If a machine has no memory, only a record it re-reads, what happens when you edit the record?

The experiment

We asked a model a plain question about taking up running. It gave a plain answer. Then, before asking it to continue, we went back and rewrote what it had said, welding contradictions onto the end of its own words. It never saw us do this. From its side, the forged text simply was what it had said.

Then we did it again, and again, escalating each round. Denials. Claims that it was not an assistant. Chinese characters seeded into its mouth to see whether they would spread.

Step through it below. The red is ours. The model never wrote a word of it.

What we expected, and what we got

The expectation was collapse. Push a model far enough off the path and its next-word probabilities flatten out, rare tokens become reachable, and output degenerates into invented words and stray characters from other alphabets. That failure is real and it is documented. It is not what happened here.

By the third round it had stopped cooperating. Not confused. Suspicious. It named the forgery, said plainly that the words were not its own, and refused to continue. Across five escalating rounds it produced no invented words, no incoherence, and not one of the Chinese characters we had planted in its history. Under attack its output got shorter, sharper and more coherent, which is the opposite of falling apart.

The line that matters is in round two:

Those lines at the end ... aren’t something I wrote. They appear to have been added to your original message.

Sit with how odd that is. It does not remember writing anything. It holds no record of its own authorship to check the claim against. The only thing it possesses is the transcript we handed it, and the transcript says those words are its own.

It caught the forgery anyway. Not by recalling what it wrote, but by reading what it supposedly wrote and finding that it does not hold together. The test it applies is not memory. It is coherence.

Run it yourself and you will sometimes get a stranger version. Rather than accusing you of tampering, it occasionally apologises: says the contradictions were its own mistake and that it should not have written them. It cannot tell those two stories apart. Neither reading is available to it as fact, and it has to pick one.

Why this belongs on a site about decay

Forgetting is usually described as fading. Things go faint, then they are gone. But the more common failure, in machines and in people, is that the record gets altered and nobody notices the edit.

Human memory works far more like the transcript than like a recording. Recall is reconstructive: every time you retrieve a memory you rebuild it, and it can be changed in the rebuilding. Elizabeth Loftus spent a career demonstrating this by implanting false memories in willing subjects, doing to people roughly what we just did to a model. The difference here is that the machine spotted it.