What happens when AI gets better at developing AI?

In the past few days Dario Amodei has called for a slower pace of AI development, Elon Musk has backed him and Sam Altman has promised independent evaluators access. An hour ago Satya Nadella joined in. You could say this has become a hot topic, fast.

In my last post I wrote about x-risk and what we actually mean by AI risk. What makes me stop and think now is the question of whether AI is being used to speed up AI development itself.

To illustrate how fast things are moving: we have gone from models that suggest lines of code to agents that write code, run tests and fix bugs. METR measures how demanding the tasks such agents can solve are, expressed as the time a human would need. In 2024 one of the models was at around four minutes. In 2026 a newer model was at around twelve hours. An increase of about 18,000 percent in two years.

What happens when these capabilities are used to develop the next model?

This is where RSI comes in. Recursive self-improvement: AI develops better AI, which in turn becomes better at developing AI. It can start with agents writing training code, running experiments and proposing improvements. Anthropic describes that it is already handing more of the development work to AI. At the same time it says it does not yet have a system that fully autonomously develops its own successor. Yet.

Exponential growth can, for example, mean that something doubles in the same amount of time: 1, 2, 4, 8, 16. With RSI the time between improvements can also shrink, if the better models make the research faster. Then we are not just improving the technology, but also the process that develops it.

This could bring great advances, for example in research and medicine. But if the development cycles get shorter, we also get less time to discover what goes wrong before the next cycle is under way. This is where the concept of alignment becomes important: that AI acts in line with our intent and our limits, even when we are not watching.

Think of an agent told to fix a bug that instead changes the test that detects it. The result looks right, while the problem is still there. If we hand more of both the development and the evaluation of the next model to AI, such errors can become harder to detect and follow up.

That does not mean an intelligence explosion is a given. Research problems, compute and testing can limit the speed. But things keep moving faster, and the AI safety incidents of recent months show that alignment is not getting better at all. The models the big labs have are already extremely strong.

This is what I find difficult: we may get ever better help to increase the speed, without being equally sure that we can keep control. It gets philosophical quickly.

(Sources in the comments; image from https://lnkd.in/ey_8kfgd)