The Collision Between AI Caution and Self-Improving Models Sparks Debate
Nadia Petrov | AA
Global calls to slow AI development face challenges as technology advances toward self-improving systems, prompting concern from leaders like Pope Leo XIV and Elon Musk over risks to jobs, misinformation, and societal control.
Calls for a global pause or slowing in the development of artificial intelligence (AI) are meeting stiff resistance as technological advancements forge ahead at a rapid pace. Among these advancements is the increasingly autonomous nature of AI systems, which are bringing humanity closer to a future where AI builds its own successors with minimal human oversight.
US tech company Anthropic has heightened these concerns with a recent report suggesting that autonomous AI development may be nearer than previously thought. The company revealed that more than 80% of the code for its Claude chatbot is now generated autonomously by the model itself. This marks a significant leap from early 2025, when the proportion was reportedly in the single digits.
What did the Future of Life Institute's letter warn about AI?
The March 2023 letter from the Future of Life Institute warned that AI approaching human-level intelligence could cause massive job losses, misinformation, and a potential loss of human control over civilization.
Using a staged AI development framework, Anthropic outlined that current technology is now at stage four of five. In this penultimate stage, models can independently execute code and assign tasks to other systems. The fifth stage, however, looms large as the point where autonomous agents could design and train future iterations of AI entirely on their own , a development some critics argue could make halting or controlling AI’s progress nearly impossible.
Warnings from Leaders Across Fields Highlight Risks
Despite these advancements, voices from academia, industry, and public institutions continue to raise alarms about the risks of unchecked AI development. A landmark warning came in March 2023 from the Future of Life Institute. In an open letter signed by prominent figures, including billionaire entrepreneur Elon Musk, the institute called for a six-month moratorium on training AI models more powerful than GPT-4.
The letter emphasized the risks of AI systems achieving human-level intelligence and warned of consequences such as widespread job displacement, a surge in misinformation, and the loss of human control over key aspects of civilization. 'We need to ask ourselves if humanity can still operate with these systems unchecked,' the letter's authors urged.
Adding to ethical calls for restraint, Pope Leo XIV made an appeal for a halt to AI development in his May encyclical Magnifica Humanitas. The Pope stressed the imperative of removing AI from the race for military and economic supremacy, framing the issue as a moral concern for global stability.
Such warnings underscore a sobering truth: technologies that could help or harm societies on an unprecedented scale are developing faster than corresponding ethical and regulatory frameworks can keep up.
A Financial And Competitive Juggernaut
One of the core challenges to slowing AI development lies in the colossal investments pouring into the sector. According to data from Gartner, global spending on AI is projected to grow by 47% in 2026, reaching $2.59 trillion, and is expected to rise further to $3.49 trillion in 2027. These figures highlight the momentum of an industry driven by intense economic and strategic competition among nations and private enterprises.
The sheer scale of these investments renders global cooperation on AI regulation complex, as even a temporary pause could mean competitive disadvantages for those who adhere to such measures.
Why Cities and Urban Leaders Must Navigate AI's Impacts
The rapid progress in AI development , particularly the prospect of self-improving systems , poses unique challenges and opportunities for urban leaders and municipal governance. Cities serve as testing grounds for many AI applications, from traffic management to public safety, but they also stand on the front lines of safeguarding jobs and combating misinformation that proliferates through advanced systems.
As investment in AI crosses trillions of dollars, city officials must grapple with policies that encourage local innovation while ensuring responsible deployments. Urban leaders should also advocate for collaboration between private developers and public regulators to prioritize ethical and societal considerations. For mayors and city councils, the role of partnerships, public communication, and education in demystifying AI risks will only grow in importance as these technologies continue to evolve.
Questions That Matter
What did Anthropic reveal about its Claude chatbot?
Anthropic disclosed that over 80% of the code for its Claude chatbot is now autonomously generated by the model itself. This is a significant increase from early 2025, when the autonomous code generation was in the single digits.
What did Pope Leo XIV emphasize in his May encyclical?
In his May encyclical Magnifica Humanitas, Pope Leo XIV emphasized the need to remove AI from the race for military and economic dominance. He framed the issue as a moral concern crucial for maintaining global stability.