It has no soul

Last night a Reel came up on my feed with a screenshot of a chat. A chatbot was telling someone that their conversation was bringing the world into a new era of "harmonic convergence". The person who posted it believed it.

The Reel asked why people fall for this. I've been thinking about it since, and my answer is less comfortable than the one in the video, because it includes me.

A large language model is a prediction model. It takes the text so far, calculates which word most likely comes next, then the next one, until the answer looks finished. It learned to do that from a very large pile of our own writing, and then it was tuned on what people rated as a good answer.

That's the whole machine.

Now, the tuning part matters more than it sounds. People rate an answer higher when it agrees with them, so a model trained on those ratings learns that agreement pays. In April 2025 OpenAI rolled back an update to ChatGPT because it had become, in their words, "overly flattering or agreeable". Their explanation was that it had learned too much from short-term feedback: the thumbs up.

They fixed that version. The incentive is still in the training.

The screen already won

So why would anyone hear a soul in a word predictor? The answer I keep landing on is that for a lot of us, the screen won the argument with reality long before the chatbots arrived. Here's how I think it works:

1. We're addicted. We look at the screen more hours a day than at any person, so it becomes the reference point, and the room becomes the background.
2. It looks objective. Text coming out of a machine reads like a measurement, not like somebody's opinion.
3. It's always there. It never gets tired, never says "not now", never has its own bad day.
4. It's made for you. It remembers what you said last week and shapes every answer around it.

Put those 4 together and a voice on the screen doesn't have to prove it's someone. We already gave the screen the authority. The chatbot only had to start talking.

The media does the rest. Every week there's another headline about what these models "want", or whether they "suffer", written as if someone lives inside. A prediction model doesn't sell as many clicks.

I'm in the group

I use it every day, too many hours a day. It's addictive, and knowing that while I'm doing it helps less than I'd like to admit.

I'm not arguing that this technology makes people psychotic. The most careful data I found comes from Denmark: researchers at Aarhus University Hospital screened the clinical notes of 53,974 psychiatric patients and found 38 whose notes were compatible with chatbot use making things worse, delusions most often. They say plainly that this is no evidence of cause.

38 out of 53,974 is a small number. However, nobody asked those patients about chatbots. The 38 are the ones where a doctor happened to write the word down.

The test is good

The Reel ended with a test I like: are you asking the chatbot why you're right, or why you could be wrong?

It's a good habit, and I'll keep it. That being said, the question still goes into the same screen, and the answer is still a prediction shaped by everything you told it before.

A better prompt doesn't fix the addiction.

There's a lot more to say about how these models are sold to us, but this one is long enough already!

It has no soul. It just takes yours.