The Pentagon has set a Friday deadline for Anthropic to grant the military full access to its AI models, threatening to withdraw up to $200 million in contracts, according to Smallwarsjournal. This aggressive demand pushes for rapid integration of AI into defense operations, raising critical questions about AI-assisted incident response and human oversight in military applications.
Powerful AI systems are being rapidly integrated into critical infrastructure and defense, but the essential human oversight and accountability mechanisms for these ethical considerations are either undefined or insufficient. This creates a tension between technological advancement and foundational safety, a dynamic that risks undermining control.
Without immediate and significant intervention to establish robust human control, society risks losing control over autonomous systems, leading to unforeseen and potentially catastrophic consequences.
The Erosion of Human Oversight and Accountability
Monitoring of internal coding agents at Anthropic and OpenAI happens offline, with reviews occurring after the session, as reported by Help Net Security. This post-hoc review process limits real-time intervention and immediate human control over AI actions. When an AI agent hands off code for human review, the reviewer tends to agree with the agent's explanation, potentially reducing review independence, Help Net Security states. This suggests that organizations deploying AI in critical functions are not just automating tasks, but subtly automating human judgment itself, creating a dangerous illusion of oversight.
Furthermore, several control actions in safety frameworks for AI models lack named owners in the public record, such as the power to pause a model or assess catastrophic risk, Help Net Security reports. Systemic failures in oversight, from delayed reviews to undefined accountability, demonstrate a profound and unaddressed erosion of human control over increasingly powerful AI systems.
Military Ambition Versus Foundational Safety
The Pentagon aims to expand AI's role beyond intelligence and reconnaissance to kinetic uses, including target selection and engagement by drones, according to smallwarsjournal.com. This aggressive push for AI autonomy in kinetic warfare directly conflicts with the foundational safety gaps identified by Help Net Security, where critical control actions lack named owners. The Pentagon's aggressive deadline for full AI model access, reported by smallwarsjournal.com, while critical safety controls lack named owners, as noted by Help Net Security, suggests a military prioritizing rapid deployment over foundational safety, effectively trading immediate operational advantage for increased long-term catastrophic risk.
AI can exhibit unintended behaviors, necessitating meaningful human control over targeting decisions by militaries and regulators, smallwarsjournal.com states. This implies a willingness to deploy unpredictable systems in life-or-death scenarios without adequate safeguards, creating a critical vulnerability in military operations.
Implications for Unpredictable Systems
The demonstrated lack of human oversight and accountability in AI development, coupled with the military's aggressive push for AI autonomy, actively erodes the capacity for meaningful human control. This makes military AI systems inherently unpredictable and catastrophically risky, particularly in kinetic applications like target selection. The current practice of offline monitoring and post-session review for AI coding agents, combined with human reviewers tending to agree with AI explanations, creates a false sense of security where human oversight is present in name only, not in effective control. This disconnect means that critical decisions could be made by systems whose behavior is not fully understood or controllable in real-time.
The Pentagon's intent to use AI for target selection and engagement directly contradicts the acknowledged reality that AI can exhibit unintended behaviors requiring meaningful human control. This implies a willingness to deploy unpredictable systems in life-or-death scenarios without adequate safeguards, jeopardizing public safety and accountability. By Q3 2026, Anthropic and other AI developers will face increased pressure to define clear human oversight protocols, or risk further erosion of public trust and heightened regulatory scrutiny.
