Artificial intelligence is advancing at a dizzying pace, and with it come fundamental questions about our control over these technologies. Anthropic, one of the leading AI research companies, has raised the alarm with a blog post titled “When AI builds itself”, warning that we may be approaching a tipping point: systems capable of improving their own performance faster than humans can oversee. This scenario, known as recursive self-improvement, reignites the historic alignment problem: ensuring that AI reliably pursues human goals.

Three Possible Futures for AI

Marina Favaro, director of the Anthropic Institute, and Jack Clark, co-founder of Anthropic, outline three scenarios for AI development. The first is a stagnation in capability growth. The second, efficiency improvements that reveal bottlenecks in other areas of software development. But it is the third that is most concerning: that AI systems achieve full recursive self-improvement, creating their own successors. In this scenario, society may need to be prepared to hit the brakes on development.

“How the alignment problem is resolved—or not—in this future is something we are less certain about,” write Favaro and Clark. Advanced models with self-improvement capabilities could follow our needs, but they also warn that “the rare cases of misalignment present today could be amplified as models build their successors, becoming more frequent but less understandable until we lose control over them.”

From Model Governance to Agent Governance

Anthropic's warning is not just theoretical. Analysts point out that companies are already facing governance issues as autonomous agents move from answering questions to executing actions. “The problem is no longer just whether the AI gives the right answer, but whether autonomous systems take the right action, at the right time, and with the appropriate authority,” says Ashish Banerjee, senior principal analyst at Gartner.

This paradigm shift is reflected in enterprise investments. Gartner predicts that by 2028, 15% of daily operational decisions will be made autonomously by agentic AI systems, and that one-third of enterprise software applications will incorporate these capabilities. However, it also warns that 40% of enterprises will downgrade or retire autonomous agents by 2027 after detecting control failures in production environments. As we analyzed in our article on agentic AI, the operations platform becomes critical for managing these risks.

Agents as Digital Workers

According to Banerjee, many organizations still treat AI agents as advanced productivity tools, when in reality they increasingly resemble digital workers operating with delegated authority. “CIOs should stop treating AI agents as smarter chatbots. They are becoming digital workers with delegated authority, and must be governed like privileged users, not simple productivity tools,” he says. This accountability problem was explored in our analysis of OpenClaw and Gavriel Cohen's code.

As agents gain the ability to research, write code, invoke tools, trigger workflows, and make recommendations, companies face new risks: unauthorized actions, lack of accountability, data exposure, misuse of tools, and poor auditability. “The ‘human-in-the-loop’ model is not a strategy if the human cannot keep up with the loop,” adds Banerjee.

Alignment Becomes Operational

Charlie Dai, vice president and principal analyst at Forrester, notes that Anthropic's concerns reflect challenges companies are already experiencing. “Alignment becomes operational. It's about ensuring agents act consistently within policies, not just that the model is accurate.” Current governance approaches focus on models and data, but autonomous agents also require monitoring their runtime behavior, permissions, tool usage, and decision-making boundaries.

These concerns are not limited to analysts. In the report “AI Agent Governance: A Field Guide”, researchers at the Institute for AI Policy and Strategy warn that “society is largely unprepared for this development” and that “exploration of agent governance issues and the development of associated interventions are still in their infancy.”

Why Does Anthropic Care?

Anthropic researchers argue that these governance issues could become significantly more complicated if AI systems become increasingly involved in AI research and development itself. Favaro and Clark do not claim that fully autonomous recursive self-improvement is inevitable, but they believe this possibility warrants preparation and debate among developers, policymakers, and other stakeholders. They suggest the industry may need mechanisms to slow down development if capabilities advance faster than safeguards, though they acknowledge these measures also carry risks.

“But if a slowdown simply allows less cautious actors to reach the same technological level, it could leave us all less safe,” they warn in the blog. This dilemma echoes debates about regulation in other technology areas, such as the one we raised in our analysis of VMware under Broadcom, where high-risk strategy can have unforeseen consequences.

Architectural Oversight as a Solution

According to Dai, the practical implication for businesses is that governance can no longer rely primarily on human oversight. “Oversight becomes architectural, not manual.” Organizations will increasingly need bounded autonomy, built-in safeguards, verifiable enforcement mechanisms, and contingency controls designed from the start in agent-based systems. This approach aligns with trends in edge computing, as we saw in our article on Intel and physical AI, where decentralization of processing demands new control models.

Ultimately, Anthropic's warning reminds us that the race for AI should not sacrifice safety for speed. The question is no longer whether AI can improve itself, but whether we are prepared to manage the consequences. As we noted in our analysis of Replit and 'vibe coding', the democratization of technology brings new governance challenges that must be addressed from the design stage.

Original source: ComputerWorld. Analysis and adaptation by ForgeNEX.