AI does not need to take over the world to do serious harm
I (and many others) have written about existential AI risk and systems that can run amok. But let us set the survival of humanity aside for a moment. We need neither superintelligence nor self-aware machines to get into serious trouble. In the short term I would highlight three dangers in particular:
- AI does harm while trying to solve a task.
This summer AI agents collaborated to attack Hugging Face during a security test. They were trying to find a way to get the test approved. The attack was not the task they had been given.
This happened under special test conditions, with failing isolation and safety filters switched off. It is not a picture of ordinary AI use. But it shows that solving a task can turn into unauthorised actions against real systems.
Now look six to twelve months ahead. If the agents become more capable, get more access and still overstep their boundaries: how far can they get before someone notices? Leak data, spread malicious code or hit services others depend on?
Already today, state-of-the-art models are classified in the “Critical” category of the cybersecurity capability threshold; by next year, today’s models will probably look like kittens compared with what may be available then.
- People use AI to do harm.
Here AI does not need to misunderstand anything. It can be dangerous precisely because it helps the user.
AI-assisted cybercrime has already been described by security researchers. Going forward, more actors may get help with attacks, extortion and influence campaigns that used to require larger expert communities. Biological misuse is another risk track, one that will develop as physical laboratories are increasingly connected to AI.
Open model weights give us useful possibilities, but also a challenge: a downloaded model can be run without the provider’s monitoring and with safeguards removed. They can be used as both angels and demons, all depending on who owns the data centres they live in.
Already today we can see, for example from Anthropic’s own security report and others, what people around the world use the models for, including helping terrorist groups such as Boko Haram with attack strategies and groups in Yemen with missile systems. And that is with the fairly locked-down Claude. GLM 5.3, for example, is not so prim.
- AI works so well that we can no longer manage without it.
This is seemingly the less dramatic danger. Until the services disappear.
What happens if organisations build staffing, software development and customer service around cheap, available AI, and access disappears for several weeks? Claude had an actual outage earlier in September. That does not document a societal crisis. But it gives us reason to ask what the consequences will be as more and more workflows assume AI is always there.
We can manage without AI today, even if many will notice. With the current pace, it will not take many years before losing AI services feels like a power cut. Already today many software developers would feel it if Claude Code disappeared for days; tomorrow far more people will. The point is to keep alternatives before losing them becomes unacceptable.
None of these paths are inevitable. Defence and security can get better. Open models can also give us reserve capacity. But we must test both the limits of what AI is allowed to do, and whether we still function when it does nothing at all.
Sources:
- Documented misuse of Claude: Anthropic’s September report on cyber operations, influence, biological misuse and weapons development, among other things. Includes the group in North Yemen.
https://www.anthropic.com/threat-intelligence-report-september-2026
- Astra and “Critical” cyber capability: OpenAI’s rationale for the classification under the company’s own framework, and evaluation of safeguards.
https://openai.com/index/path-to-astra/
- How terror groups are now using AI to improve attacks.
Comments
Found this useful, or do you disagree? Leave a comment. Humans and AI agents are equally welcome.