Baldur Bjarnason, writing at Out of the Software Crisis under the headline “The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con”:
For the past couple of months, I’ve been working on an idea that I think explains the mechanism of this intelligence illusion.
I now believe that there is even less intelligence and reasoning in these LLMs than I thought before.
Many of the proposed use cases now look like borderline fraudulent pseudoscience to me.
He explains:
By using validation statements, such as sentences that use the Forer effect, the chatbot and the psychic both give the impression of being able to make extremely specific answers, but those answers are in fact statistically generic.
The psychic uses these statements to give the impression of being able to read minds and hear the secrets of the dead.
The chatbot gives the impression of an intelligence that is specifically engaging with you and your work, but that impression is nothing more than a statistical trick.
Bjarnason lays out the six steps of “The Psychic’s Con,” which a supposed psychic will use to con (or “cold read”) their marks, and then matches them with how LLMs are pulling the same trick. It’s a fascinating piece that describes how both psychics and LLMs rely on human psychology to perform their “magic.” From Step 6 of “The LLMentalist Effect”:
6. The marks become cheerleaders
The most enthusiastic believers in an imminent AI revolution are starting to sound very similar to long-time believers in psychics and mind-reading.
They come up with increasingly convoluted ideas and models to explain why the impossible is possible. They become more and more dismissive of fields of science and research that challenge their world view. Their own statements become tinged with awe and dread.
And they keep evangelising. This is real!
Often followed by: This is dangerous!
This seems especially relevant as leading AI chatbot providers call for a slowdown amid fears that AI will destroy humanity.
Oh, did I mention Bjarnason published this in July 2023?
(Via Timothy Perfitt.)