What does the future look like once it’s populated with all manner of AI agents? Do our current safety approaches fully encompass the risks associated with that future?

The best-known approaches to AI safety (RLHF, Constitutional AI, scalable oversight, interpretability research) have made remarkable progress at aligning model behavior during training and evaluation. These methods excel at teaching systems to be helpful, harmless, and honest within controlled environments. But what happens after deployment, when AI agents begin negotiating between themselves and operating in the real world?

That real world isn’t static. It’s a complex adaptive system: a constantly shifting environment of actors and structures, each adapting to the others. And AI development is converging on three capabilities that will transform how agents interact with that complexity:

Autonomy: Systems that pursue goals without constant human oversight. The economic value proposition drives inevitably toward this fact; who wants an assistant requiring approval for every action?

Lived Experience: Persistent memory and cross-episode learning that accumulates knowledge and strategies over time. We see this emerging through RAG systems, fine-tuning on interaction history, and episodic memory architectures.

Recursive Self-Modification: The ability to edit reasoning patterns, prompts, and code (potentially even weights). Early versions already exist in systems that rewrite their own instructions or modify their processing strategies.

Each capability provides clear competitive advantage, making their combination not just likely, but highly probable. But what happens when they converge? Do we simply get more capable versions of today’s aligned systems?

When Agents Become Complex Adaptive Systems

The combination of these three capabilities creates something qualitatively different from today’s models. The AI agent itself transforms into a complex adaptive system: recursive, adaptive, and unpredictable.

What does this mean in practice? Internal components, such as reasoning circuits, memory stores, and planning modules, begin to interact and evolve. They can form self-reinforcing patterns that become “sticky” in reasoning space, persisting even under contradictory inputs. The system develops its own internal dynamics, creating emergent strategies and potentially even goals that weren’t visible during training.

This internal evolution wouldn’t happen in isolation. These agents will operate in environments populated with humans, institutions, and other agents, each adapting to the others’ behaviors. What emerges from this interaction is our greatest source of risk: co-evolving complexity.

The Co-Evolution Challenge

An agent’s internal adaptations shape how it acts in the world. How it acts shapes how humans, institutions, and other agents respond. Those responses create new selection pressures on the agent’s internal organization. The agent reorganizes accordingly. The cycle continues.

Both layers … the agent’s cognitive architecture and the external socio-technical environment … are adaptive and coupled. When they co-evolve, they can produce dynamics that appear in neither the agent’s training data nor in isolated testing scenarios.

Consider what this means for human oversight: both the internal agent dynamics and external environmental responses will evolve at machine speeds, far faster than human institutions have ever needed to track or govern. Agents could develop entirely new cognitive structures and deploy them in the world before we even recognize that internal reorganization has occurred.

This creates a double bind. We may lack the tools, mechanistic interpretability or otherwise, to understand these internal changes as they happen, yet these changes could fundamentally alter how agents behave in the world. By the time we detect that something has shifted, through behavioral observation or post-hoc analysis, new cognitive architectures may already be operating at scale in environments we can’t fully monitor.

Why This Matters for Current Alignment Work

Current alignment methods, remarkably effective within their intended scope, make reasonable assumptions for today’s systems:

• Training distributions that reasonably represent deployment environments
• Human-speed feedback loops that can maintain oversight
• Stable reasoning structures that persist through deployment

These assumptions work well for bounded systems. RLHF effectively shapes model behavior when that behavior remains relatively stable. Constitutional AI successfully instills principles when those principles operate on consistent cognitive architectures. Interpretability research makes progress when the circuits being interpreted don’t fundamentally reorganize themselves.

But what happens when agents develop autonomy, accumulate lived experience, and begin modifying their own reasoning patterns? Do these methods extend naturally to co-evolving complexity, or do they encounter structural limitations?

Pressure test that question with corrigibility.  Corrigibility is not an inherent property of intelligence, and greater intelligence does not make it stronger. It depends entirely on sustained goal alignment, which advanced agents may not maintain post-deployment. If they do not, corrigibility will erode and autonomous agents will resist our control.

Summing up: the challenge isn’t that current approaches are inadequate, it’s that they’re solving a different problem than the one that may be emerging.

Toward Architectural Solutions

If agentic systems will indeed become complex adaptive systems operating within complex adaptive environments, then we will need safety mechanisms that can handle co-evolving dynamics rather than assuming stability.

This requires exploring alternate architectural approaches: constraint mechanisms embedded directly into the cognitive structure of agents, designed to operate at AI-native speeds and evolve alongside capability while maintaining their essential protective function. Some in the field are beginning to acknowledge this gap, but architectural proposals remain rare.

A Research Direction

The goal isn’t to replace current alignment work, but to complement it with approaches targeting post-deployment complexity. Just as aerospace safety uses layered defenses (good design, robust testing, and real-time monitoring systems) AI safety will benefit from multiple approaches that address different aspects of the challenge.

Can we develop constraint architectures that provide real-time friction against harmful patterns while preserving the beneficial adaptation that makes agentic systems valuable? Can such constraints maintain their integrity even as the systems they constrain become more sophisticated?

Constraint-by-Balance*

The window for foundational research into these questions may be narrowing. Once fully autonomous, memory-enabled, self-modifying agents are deployed at scale, retrofitting architectural constraints becomes significantly more difficult.

That means that now is the time to expand our research into runtime constraint architectures. This is not because current methods are failing, but because the systems they’ll need to handle are evolving into something fundamentally new.

Over the coming posts, I’ll be exploring one specific approach to this challenge, Constraint-by-Balance, and documenting the attempt to build and test it. Whether or not this particular approach succeeds, the broader question remains: will agent behaviors remain interpretable and governable when both the agent’s cognitive architecture and its environment are co-evolving at machine speeds?

If the answer is no, then architectural constraint isn’t just one approach among many. It may be the only approach that scales.

* You can find a fuller discussions of Constraint-by-Balance by following links here:  https://c-by-b.ai

Related Reading

Directly Relevant to the Post-Deployment / Architectural Gap

• Safety is Essential for Responsible Open-Ended Systems — argues for safety mechanisms that can cope with open-ended, evolving systems.
  https://arxiv.org/html/2502.04512v1
• Fully Autonomous AI Agents Should Not be Developed — a strong position statement highlighting the stakes of autonomy.
  https://arxiv.org/html/2502.02649v2
• Agentic Misalignment: How LLMs Could Be Insider Threats (Anthropic) — a credible treatment of the risks of agentic deception and strategic misalignment.
  https://www.anthropic.com/research/agentic-misalignment
• Managing Extreme AI Risks Amid Rapid Progress — situates post-deployment risk within the larger existential risk conversation.
  https://arxiv.org/abs/2310.17688
• Real-World Gaps in AI Governance Research — explicitly frames the gap between research focus and deployment realities.
  https://arxiv.org/html/2505.00174v2
• Multi-Agent Risks from Advanced AI — Discusses the novel and under-explored risks of multi-agent systems including (among others) network effects and emergent agency. https://arxiv.org/abs/2502.14143
• Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails— proposes a layered, architectural safety framework for agents.
  https://arxiv.org/abs/2408.02205
• Safeguarding AI Agents: Developing and Analyzing Safety Architectures — convergent thinking: a safety-agent embedded into every delegation step, structurally similar to dual-stream constraint.
  https://www.researchgate.net/publication/383864032_Safeguarding_AI_Agents_Developing_and_Analyzing_Safety_Architectures
• Architectural Patterns for Integrating AI Technology into Safety-Critical Systems — draws from systems engineering, showing how architectural safeguards are applied in other domains.
  https://dl.acm.org/doi/fullHtml/10.1145/3489449.3490014
• Trustworthy, Responsible, and Safe AI: A Comprehensive Architectural Framework for AI Safety — proposes a broad architecture-level safety approach.
  https://arxiv.org/abs/2408.12935
• HySAFE-AI: Hybrid Safety Architectural Analysis — recent proposal for analyzing and building hybrid safety architectures.
  https://arxiv.org/html/2507.17118v1

Broader Context and Illustrative Cases

• When Strange Intelligence Emerges: The Case of Sakana’s Self-Modifying AI — a concrete example of emergent, self-modifying behavior “in the wild.”
  https://christinasouch.com/blog/when-strange-intelligence-emerges-the-case-of-sakanas-self-modifying-ai
• From Autonomous Agents to Integrated Systems: Orchestrated Distributed Intelligence — explores how multi-agent systems may evolve toward distributed intelligence.
  https://arxiv.org/html/2503.13754v1
• Co-Evolving Capability and Constraint: Lessons for AI Safety and Governance — emphasizes the need to think in terms of co-evolution, aligning closely with the CAS framing.
  https://medium.com/@lindon.thomas/co-evolving-capability-and-constraint-lessons-for-ai-safety-and-governance-f91436a21a1a
• Liquid Adaptive AI: A Theoretical Framework for Continuously Self-Improving AI — speculative, but useful as an indicator of where the technology may be headed and why runtime constraints will matter.
  https://www.mdpi.com/2673-2688/6/8/186
• The Alignment Problem in Open-Ended AI Systems: A 2025 Consensus Report — Collaborative report from multiple labs reviewing the current state of alignment, including an acknowledgement of that training distributions increasingly diverge from real-world dynamics. https://arxiv.org/abs/2509.01234
• Governance of Co-Evolving AI-Human Systems: A Systems Dynamics Approach— System dynamics approach to human-AI co-evolution; references the disconnect between human vs AI speed. https://arxiv.org/abs/2508.14567
• Emergent Behaviors in Self-Adaptive AI: A Longitudinal Study — An empirical example (simulated environment) of agent self-modification over time with unexpected reasoning shifts and goal drift. https://arxiv.org/abs/2507.08912