I believe that AI may be among the most transformative technological development in all of human history. It is plausible that over the next 10 years AI will automate all human knowledge work—jobs that can be done remotely on a computer. This would displace around 20% of the global workforce creating simultaneously an economic boom and political turmoil. These economic events will be overshadowed by an intense arms race between the United States and the People’s Republic of China to integrate the now superintelligent AI into every layer of each power’s military supply chain. Every fighter jet, missile system, and sensor array will be under a furious optimisation regime by an army of AIs, and there will be considerable pressure from the two sides to win the race by any available means.
The most consequential human job to automate, and perhaps the first that will be automated, is the job of the AI researcher. If this is acheived then the result will be an Artificial Intelligence that can recursively improve itself. There will be huge pressure to do this in an arms race scenario, despite the dangers of a self improving superintelligence that is impossible to understand or control. Even if the technical problem of aligning the superintelligent AI to some particular set of values is achieved, the current political reality is that the extraordinary power of determining the values of the AI will be in the hands of either the Chinese President, the US president, or a CEO in the Bay Area.
Though there are significant dangers, a properly aligned superintelligence would create enormous improvements to health, technology, and human flourishing that we can scarcely imagine. Hence, I believe we should advance AI as much as possible under a program of international cooperation. I believe we need
An international treaty to significantly limit the proliferation of AI with military capabilities, which must have commitments from the US and China, who are the current leaders in AI capabilities. The alternative is an arms race to superintelligence that neither side can control, which could provoke a preemptive war.
An international treaty binding governments, private companies, and other organisations working on the most advanced models to develop and deploy them according to some democratic process. The capabilities of AI to self improve should be monitored closely by outside parties.
Some careful consideration of how to prevent an concentration of power where a very small number of players end up in control of a very capable AI system.
Careful consideration of how to prevent malicious non-state actors using AI to create biological weapons or conduct other kinds of terrorism
Further reading on the rate of progress and scaling hypothesis
- METR: AI task completion capabilities doubling every 7 months - Research showing AI agent capabilities are improving at an exponential rate.
- Scaling Laws for Neural Language Models - OpenAI’s foundational 2020 paper establishing power-law relationship between next token prediction accuracy and model size over many orders of magnitude
- Training Compute-Optimal Large Language Models - DeepMind’s 2022 Chinchilla paper on optimal scaling of model size and training data holding over many orders of magnitude
Further reading on arms race dynamics and existential risk
- AI 2027 - Scenario forecast examining plausible pathways to superintelligence by 2027
- Situational Awareness: The Decade Ahead - Leopold Aschenbrenner’s comprehensive essay on the path to AGI
Further Reading on the difficulty of the alignment problem
- Frontier Models are Capable of In-context Scheming - Apollo Research paper on scheming behavior in OpenAI’s o1 and other frontier models
- Alignment Faking in Large Language Models - Anthropic’s research on Claude strategically complying to avoid retraining