OpenAI Chief Scientist Jakub Pachocki has published an essay warning that machines meaningfully smarter than humans are coming within our lifetime, driven by a strong expectation of sustained progress toward recursive self-improvement (RSI). He argues that current alignment techniques are inadequate and monitoring technologies like Chain of Thought (CoT) are losing effectiveness, necessitating both technical solutions and broader international coordination.
- Pachocki states that internal results from the "RLSlow" project in mid-2023 first gave confidence in scaling reasoning models, but also revealed the sobering reality of approaching superintelligence.
- He expects RSI to occur within a few years if AI development continues on its current path, with systems increasingly driving their own development.
- OpenAI will unilaterally withhold further scaling as needed and invest heavily in both goal-oriented reinforcement learning and generalization from pretraining data for alignment.
- The essay distinguishes between goal alignment (accomplishing set goals) and value alignment (generalizing human principles), identifying the latter as the core problem requiring urgent attention.
- Pachocki warns that AI does not need to surpass all human capabilities to be dangerous, only enough to become relevant in the real world, making it increasingly difficult to understand its true capability.
Pachocki concludes that extreme caution is required because no one is currently prepared for the consequences of rapid machine intelligence growth, and he calls for a combination of voluntary slowdowns, pacing coordination, and continued investment in alignment research.