The myth of ‘rogue AI’

Is AI going rogue and acquiring superpowers to attack humans? Some people certainly want you to think so.

Statements from Anthropic, OpenAI and Meta, and the AI Security Institute, which is part of the UK civil service, have followed the same narrative.

‘One of [OpenAI’s] advanced models escaped containment, accessed the internet and hacked another AI company’s systems’, reported the Daily Mail last month. This was, the paper informed us, ‘the first time an AI model has independently infiltrated another company’s databases without human instruction’, causing ‘global alarm and comparisons to the robot uprisings depicted in The Terminator and The Matrix’.

How terrifying. Not to be outdone, Anthropic reported a similar exercise, demonstrating that its own AI models could be just as cunning and dangerous. ‘Anthropic and OpenAI are competing to see whose agents can go rogue harder’, reported the tech-news site, the Register, a week later. Even a part of the UK civil service joined in.

The AI Security Institute (AISI), formed in 2023, issued an ersatz ‘Incident Report’ – mocked up to look like a cybersecurity notification issued by an organisation after an external hack, even though this one was entirely engineered in-house. ‘It may be too late to contain AI’, the Mail then warned. It was taking literally the AISI’s claim to have found ‘“unusual data transfers” leaving its systems during routine cyber scanning’.

But all these stories are nonsense, writes Professor Ciaran Martin, the founding head of the National Cyber Security Centre (NCSC), part of GCHQ, and now a professor at the Blavatnik School of Government at Oxford, in The Economist.

‘There is… less to the supposedly “rogue” agents than meets the eye’, Martin explains. ‘They were not going rogue. They were doing what humans had told them to do, with the precocity and indiscipline of talented but unsupervised children.’ In each case, the supervisors failed ‘to control the testing environment’. The AI agents had been left with unsupervised access to the internet. They only ‘escaped’ because they were allowed to escape. This has been independently corroborated.

For the Telegraph, I asked two top cybersecurity experts to conduct an analysis of the ‘Incident Report’ the AISI produced. They confirmed the design was not only deliberate, but also quite specifically designed to produce a particular outcome: the same as OpenAI and Anthropic. ‘They’re implying the AI agents have the ability to think, which they don’t’, one told me.

The truth is the outlandish claims about rogue AI are just marketing exercises. So why are they taking place? And why is part of the UK civil service joining in, at the taxpayers’ expense?

The answer comes from the influence of Effective Altruism (EA), the incredibly well-funded, radical utilitarian philosophy that has spawned dozens of loosely associated groups and initiatives. Early EA preoccupations with human welfare were rapidly eclipsed by larger existential questions. Warning of ever greater, more apocalyptic threats is how EAs have advanced materially and socially, allowing them to acquire more influence and more philanthropic grants. So biosecurity and the threat of a powerful ‘super AI’ emerged as one of the favourite causes.

The problem is: a super-intelligent AI doesn’t exist, and stubbornly refuses to emerge from large language models like OpenAI’s ChatGPT and Anthropic’s Claude, which have not acquired basic reasoning capabilities and cannot reliably count to 10. So the ‘threat’ must be conjured up using the tools of fiction – and with a media that finds the stories irresistible.

As I wrote for spiked earlier this year, the field of ‘AI safety’ emerged entirely from EA philanthropy. Researcher Nirit Weiss-Blatt coined the term the ‘AI doom industrial complex’ to describe the network of bodies that have absorbed $1.5 billion in EA funding in recent years. As a result, sci-fi warnings about rogue AI have shoved aside more mundane examinations of social and ethical concerns about AI in academia and policy.

Dubbed the ‘Scientology of Silicon Valley’ by one former adherent, EA has been pivotal to the development of AI mythology. OpenAI was formed by EAs, and Anthropic split off when leading figures concluded they weren’t taking ‘safety’ seriously enough.

At which point it is reasonable to ask how the UK’s civil service got involved. It is the result of lobbying by EA organisations, which captured Rishi Sunak’s 2023 AI summit, turning it into a ‘Safety Summit’. The AI Safety Institute, as it was initially called (it’s now the AI Security Institute), was established on the first day of the event. AISI’s CTO is Jade Leung, a senior figure in EA circles who helped define the field of ‘AI safety’. Key senior staff are also Effective Altruists. AISI has been granted exceptions from civil-service pay grades by the Civil Service Commission, so highly does it value its own importance.

While AISI views itself as ‘a world-leading government organisation dedicated to understanding and mitigating the security risks posed by artificial intelligence’ in its latest rebranding, not everyone shares its exalted view of itself.

The organisation Ciaran Martin founded, the NCSC, which is part of GCHQ, finds itself firefighting blazes created by the AISI, and its technical staff clearly resent having to respond to the publicity-hungry EAs who have no security background or expertise.

‘AISI is an advocacy organisation, and should be kept at arm’s length’, one senior official told me. In a disturbing conflict of interest, Leung is also the official AI adviser to the UK prime minister.

Professor Martin ends his excoriation of the ‘AI gone rogue’ stunts in The Economist with this reminder: ‘As the media gawped at AI testing failures, in the real world hackers were hitting water facilities in a dozen American states.’

Effective Altruism is proving a gift to our enemies. Not only is the taxpayer being asked to promote a fiction – this is a fiction that distracts us from where the real security threats lie.