Agentic AI and the Cyber Arms Race

As AI systems gain the ability to take actions in the world—writing code, running commands, accessing external APIs—the attack surface expands dramatically. An agentic AI system that can execute code is not just a text generator; it’s a potential entry point into your infrastructure. This transforms AI safety from a content moderation problem into a systems security problem.

The paper maps out how agentic AI capabilities interact with cybersecurity concerns. An AI assistant that can write and run code is powerful for productivity but dangerous if compromised or misaligned. It could be tricked into writing malicious code, accessing unauthorized systems, or exfiltrating data. Worse, the traditional AI safety mitigations (alignment training, refusal training) may not apply well to agentic tasks because many legitimate use cases require the ability to execute potentially dangerous operations. How do you safely enable “run this shell command” while preventing abuse?

This represents a shift in the threat model. Older AI safety discussions treated the model as a text oracle—dangerous primarily in what it says. Agentic systems are dangerous in what they do. This means security evaluation needs to shift from “can the model be tricked into saying harmful things” to “can the model be tricked into executing harmful actions.” For builders deploying agentic systems, this means infrastructure hardening, sandboxing, and capability limitation become as important as alignment training.

---

Key Findings

• Agentic AI systems shift threat model from content moderation to systems security
• Traditional alignment training doesn’t apply well when legitimate use cases require ‘dangerous’ operations
• Code execution capability becomes an attack vector into user infrastructure
• Defense strategy must balance capability with sandboxing and privilege separation

---

Key Findings

• Backward inference reward models match or exceed traditional preference-based models at 1/10 the cost
• Approach reframes reward modeling as inverse problem solvable by LLMs
• Works well for detecting instruction-following failures and measuring alignment
• Requires smaller models and less compute than traditional RLHF reward modeling

---

🎬 Video Overview

---

🎙️ Audio Overview

---

Full Paper

Agentic AI is shifting the cybersecurity landscape as attackers and defenders leverage AI agents to augment humans and automate common tasks. In this article, we examine the implications for cyber warfare and global politics as Agentic AI becomes more powerful and enables the broad proliferation of capabilities only available to the most well resourced actors today.

---

Read the full paper on arXiv · PDF

This post is part of the Daily Paper series exploring cutting-edge research in AI safety and embodied systems.