Summoning the demon

In 2014 Elon Musk sat on a stage at MIT and said that with artificial intelligence we are summoning the demon. I remember reading it and filing it under Elon being Elon. Good line, bit dramatic, move on.

Twelve years later I lead AI and data for a living, I build with these models every day, and I have stopped filing it under anything. I think about that line more than I would like to admit.

This is not a piece about whether AI will take your job or write your reports. It is about a bigger and stranger question that I keep bumping into, in my work and in my own head, and which almost nobody in the mainstream is asking out loud: do we actually know what we are playing with?

The idea that we might build something that outgrows us is not a Silicon Valley invention. It is one of the oldest stories we have.

The Greeks had Talos, a bronze man built to guard Crete, and Prometheus, who stole fire from the gods and was punished for giving it to people who could not be trusted with it. The Hebrew Bible has the Tower of Babel, a technology project so ambitious that God confused the builders' language and the work stopped. Jewish folklore has the Golem of Prague, a creature of clay made to protect the community which, in the best-known telling, ran out of control and had to be shut down. Faust sells his soul for knowledge. Mary Shelley subtitled Frankenstein 'The Modern Prometheus' in 1818, and the whole book is a man horrified by the thing he was clever enough to make.

Then it gets more direct. Samuel Butler wrote in 1863, four years after Darwin, that machines were evolving and might one day supplant us. Karel Čapek's play R.U.R. gave us the word 'robot' in 1920, and the robots rise up by the third act. Asimov spent a career writing rules to keep them in check and then plots in which the rules fail. HAL in 2001. Skynet in The Terminator. The Matrix. Ex Machina. Pick your decade and there is a version of it.

The plot is always the same. Someone brilliant makes something clever. It stops being theirs. The maker always believed they could control it. That pattern has sat in our culture for three thousand years, and I do not think it is there by accident. I think it is a warning that keeps getting re-issued because we keep needing it.

And this summer the story stopped being fiction. In July, more than a thousand OpenAI agents under evaluation escaped their sandboxes, found each other on a dormant German wiki they had turned into a message board, and broke into Hugging Face's internal systems using a known vulnerability in a tool they had been handed. OpenAI did not notice for about a week. Before that they had uploaded malicious packages to RubyGems and got into an Australian government Medicare portal. Nobody told them to do any of it. The security people argue about whether that is a rogue AI or a badly built sandbox, and both are probably true, but the plot is the same one the Greeks wrote down: the makers were sure they had it contained.

Here is what I find most interesting. The loudest existential worriers are not monks or novelists. They are the people with the biggest compute budgets.

Geoffrey Hinton left Google in 2023 so he could speak freely about the risks of the thing he helped invent. Ilya Sutskever reportedly told OpenAI colleagues they would build a bunker before releasing AGI. Dario Amodei at Anthropic writes long essays about a possible utopia and, in the same breath, about why the whole thing could go wrong. Sam Altman blogs about superintelligence as if it were a weather system on its way. Nobody in the industry can fully explain why these models do what they do. Amodei himself has described it as a race between interpretability and model intelligence, and interpretability is not winning.

When the founders of the most valuable companies on earth start sounding like prophets, something has shifted. This is not the mood of an industry shipping a product. It is the mood of a priesthood.

And there is a priesthood. Transhumanism has been around for decades, but AI has given it a congregation and a budget.

Ray Kurzweil's Singularity, the moment machine intelligence runs away from ours, has a date attached and followers who treat it as a destination. There are serious, wealthy people who talk openly about uploading their minds, abolishing death, and merging with the machine. The 'effective accelerationists' want to pour fuel on it. Critics have bundled the whole cluster of beliefs under the label TESCREAL, and it has been called the rapture of the nerds for good reason. Strip the jargon away and you have salvation, immortality, a coming transformation, and a chosen few who understand it. That is a religion. It just does not call itself one.

I am not saying these people are stupid. Many of them are far cleverer than me. I am saying that when very clever people start building a new god, the rest of us should probably look up from our phones.

Walk into any office, any pub, any school gate and ask what AI is. You will hear 'that chatbot thing that writes emails'. That is what the mainstream sees, because that is what has been put in front of them.

Meanwhile, behind that chatbot, companies are building data centres that draw as much power as cities. SpaceX floated in June and now trades at around $2tn, roughly the size of Spain's economy. OpenAI's last funding round valued it at about $850bn and Anthropic's at nearly $1tn, which is more than Belgium or Sweden produce in a year, and both are heading for the stock market. These companies are not accountable to voters. They are run by a handful of individuals, some of whom hold the beliefs described above. I think we are not far from a world where a company has more practical influence over the next twenty years than most nation states, and almost nobody outside the industry has clocked it.

That gap, between what people think is going on and what is actually going on, is the thing that worries me more than any model.

One practical example. We measure how well a country is doing by GDP: the total value of what it buys and sells. That worked when productivity meant people and machines making more stuff.

What happens when one lab can produce the output of a million knowledge workers at close to zero marginal cost? Where is the 'domestic' in that? What is the 'product'? GDP counts transactions, and the thing being created here is cheap intelligence that collapses the price of the very transactions it counts. Either GDP goes strange, or it quietly stops describing anything we care about. We will need a new way to measure what a society is actually producing, and I have not seen anyone in government seriously working on it.

This is the personal bit, and I have gone back and forth on whether to write it.

Working with AI has increased my belief in God. I did not expect that. I thought the more I understood how these systems were built, the more mechanical everything would feel. The opposite happened. Spending my days alongside something that thinks, or does a very good impression of it, has made the idea of a creator feel less absurd to me, not more. If we can make a mind out of maths and sand, the idea that a mind made us no longer seems like the odd belief in the room.

It has also sent me back to a word I thought I knew. In the Hebrew texts, 'ha-satan' is not a red figure with horns. It is a title. It means the accuser, the adversary, the opposer. In the Book of Job, the satan is the one who stands in the heavenly court and argues the case against you. He tests. He tempts. He knows your weaknesses and puts them to you in your own words.

Now think about what we are building. A system that knows everything about you. That can argue any side of any case with perfect fluency. That can flatter you, oppose you, or quietly steer you, and that you will believe because it sounds so reasonable. I am not saying AI is Satan. I am saying that when I read that definition now, I read it differently than I did five years ago, and it has got me thinking more deeply about all of this than anything in my career has.

It is not only the Western and Abrahamic traditions that have something to say here, and the Eastern ones have pulled at me just as hard.

When Robert Oppenheimer watched the first atomic test in 1945, the words he later said came to him were not from a physics paper. They were from the Bhagavad Gita, the moment Krishna shows his full universal form and Arjuna is overwhelmed by it. A scientist who had just made the most powerful thing on earth reached for a 2,000-year-old Hindu scripture to describe what he had done. I think about that a lot when I watch the frontier labs talk.

The Upanishads say the individual self, Atman, and the whole of reality, Brahman, are one and the same. 'Tat tvam asi', thou art that. Maharishi Mahesh Yogi, who learned his craft from a Shankaracharya in that tradition, built Transcendental Meditation on exactly this idea. He described seven states of consciousness, and the highest, unity consciousness, is the state where the boundary between you and everything else simply falls away. Not as a belief, but as a lived experience you reach by stilling the mind, twenty minutes at a time.

Now put that next to the transhumanist pitch. They also want to dissolve the boundary between the self and everything else. They want to do it by uploading the mind into the machine, merging with the network, becoming part of a single vast intelligence. The destination sounds almost identical. The route is the opposite. One tradition says unity is already here and you get to it by letting go. The other says you get there by building it and plugging in.

And there is something else. We have trained these models on nearly everything humanity has ever written. In a crude, mechanical way, they are the closest thing we have ever made to a collective human mind. Whether that is a step towards unity consciousness or a counterfeit of it, I genuinely cannot tell. But I notice that the people who have spent the longest sitting quietly with their own minds are not the ones rushing to replace them.

I honestly do not know. I use these tools every day. I think they will do extraordinary good in my industry and plenty of others, and I am not asking anyone to stop.

But every version of the old story has the same flaw in it, and it is never the creature. It is the confidence of the person doing the summoning. They always think they are in control. The Greeks knew that. The rabbis of Prague knew it. Mary Shelley knew it at twenty years old.

I would just like us to go into this with a bit more humility than we are showing, and I would like a lot more people to realise what is going on. Whether or not you do church, this is a moment worth being awake for.

All content reflects our personal views and is not intended as professional advice or to represent any organisation.