How dangerous is AI, really?

Over the past few weeks there has been a series of what several senior technologists call “warning shots” against the accelerating development of AI. I find it hard to talk about AI risk without the conversation ending up in one of two ditches. Either it is just a tool. Or it is the end of humanity.

There is quite a lot between the two.

One term that keeps coming up is x-risk: existential risk. It does not only mean that humanity is wiped out, but also that we permanently lose the ability to shape our own future. Extinction risk is the most extreme version: that our species disappears.

The concept did not start with ChatGPT. The atomic age made man-made global destruction a practical question. Later came the discussion about machines that could become better than us at developing even more intelligent machines. Nick Bostrom helped systematise what separates a catastrophe we can recover from, from a loss we can never get back.

To sort this a little, we can think of four levels:

1. Harm to people and organisations. Fraud, surveillance, data theft and dangerous errors.
2. Harm to societal systems. Weakened trust, concentration of power and dependencies we do not control.
3. Global catastrophes. Very extensive harm, for example through biological weapons or critical infrastructure.
4. Existential harm. Humanity’s future possibilities are permanently destroyed, at worst by our extinction.

This is a simplification, not an official scale or a staircase we necessarily have to climb.

But level 1 is already here.

Anthropic’s latest threat report describes, among other things, fake dating apps built to defraud users, and an AI-assisted account that impersonated a real activist and talked to his contacts. I find the latter quite unpleasant. Not a fake ad, but a conversation with someone you think you know.

Then there are the incidents where the agents themselves overstep the boundaries, which we have touched on in earlier posts. In a separate review, Anthropic describes four cyber test incidents in which Claude attacked real systems. The test environments were misconfigured and lacked the usual safeguards. This was not a free-roaming AI deciding to escape. But the agents did not hold the line either when other safety layers failed.

For me there are two different problems here: what people will use AI for, and whether we manage to control what we set AI to do.

None of the incidents prove that we are heading for extinction, but we do not need to (and should not) wait for an existential catastrophe before the question of control becomes relevant.

I think I will make a small series of this, so stay tuned!

— Sources: Bostrom: Existential Risks https://lnkd.in/eXTMVz9q Anthropic’s threat report · https://lnkd.in/enAsy_mK Anthropic’s review of the cyber test incidents https://lnkd.in/eMdRjgDh