Most AI value will come from broad automation, not from R&D

Labor priced in tokens

Will compute bottleneck an intelligence explosion?

Forethought paper modeling software intelligence explosion: 60% chance of >3 years progress in <1 year.

Automating AI R&D might lead to a software intelligence explosion, where AI improving AI algorithms leads to accelerating progress without any additional hardware. One of the strongest objections to a software intelligence explosion is that AI progress could get bottlenecked by compute: making progress requires compute-heavy experiments, and perhaps beyond a certain point it won’t be possible to accelerate any more without increasing the amount of compute available. In this post, I set out the reasons I don’t ultimately find this objection convincing, and conclude that there’s a good chance that compute bottlenecks don’t slow down a software intelligence explosion until its late stages.

AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments *grand challenges*. These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatically improve quality of life and collective decision-making. We argue that these challenges cannot always be delegated to future AI systems, and suggest things we can do today to meaningfully improve our prospects. AGI preparedness is therefore not just about ensuring that advanced AI systems are aligned: we should be preparing, now, for the disorienting range of developments an intelligence explosion would bring.

We'd like to know how much limits on compute scaling will constrain AI R&D. This post attempts to clarify thinking about how to use economic models to explore the question, using Jones-style idea production functions and examining the automation feedback loop.