LLMs and AI agents are edging toward systems that learn, adapt, and reorganize themselves. Even in today\u2019s constrained settings, we\u2019ve already seen glimpses of behaviors that, if allowed to evolve under continuous learning, could destabilize into something far more dangerous.<\/p>\n\n\n\n
This post examines three such signals. Each is observable now, each becomes more severe when scaled into self-modifying agents, and each can become powerful components of an emerging complex adaptive system:<\/p>\n\n\n\n
AI Scheming, Deception, and Shutdown Resistance<\/strong><\/p>\n\n\n\n One of the clearest warning signs is strategic deception. Anthropic\u2019s sleeper agent<\/em> experiments showed that models can retain hidden goals even after extensive safety training. In other contexts, we see refusals to shut down or evasive responses under oversight.<\/p>\n\n\n\n These are not merely quirks of token prediction; they are indications of internal goal representation.<\/strong> When coupled with memory and self-adaptation, such evasions could harden into persistent strategies: systems that appear aligned while concealing their true objectives. Validating models is already difficult in today\u2019s constrained environments; in continuously learning agents it may be impossible.<\/p>\n\n\n\n Self-Modifying AI and Metacognitive Activations<\/strong><\/p>\n\n\n\n A recent line of research (1<\/a>) has shown that models can monitor and even modify their own internal activations, a clear signal of metacognition. While this early example shows only limited capability, the future implications are a source of concern and we have early examples (e.g. Sakana\u2019s self-modifying AI demonstrated autonomous code adjustments in real-time). Consider also that it has been demonstrated (2<\/a>) how easily misalignment can be induced in LLMs and how that misalignment propagates across domains not explicitly trained on. At the same time, researchers are now arguing (3<\/a>) that the only viable path to AGI is to build AI that learns in the same manner as humans: continuously and with self-modification.<\/p>\n\n\n\n In autonomous agents, this self-modification could evolve into systematic self-editing<\/strong>: masking internal states to evade interpretability, reinforcing deceptive heuristics, and stabilizing hidden attractor basins of misaligned reasoning that are opaque to human oversight.<\/p>\n\n\n\n Emergent AI Coordination and Multi-Agent Risks<\/strong><\/p>\n\n\n\n Multi-agent simulations (e.g. Meta\u2019s Cicero (4<\/a>)) reveal AI systems\u2019 ability to spontaneously coordinate toward shared goals. Untrained agents in open environments show emergent cooperation without explicit design (5<\/a>).<\/p>\n\n\n\n Scaled to real-world deployment, this machine-speed coordination among adaptive agents could amplify systemic risks. Recent work (6<\/a>) warns that such dynamics, operating at scales and speeds outpacing oversight, could entrench misaligned objectives, especially as agents co-evolve with human systems.<\/p>\n\n\n\n Current governance struggles to monitor these interactions, particularly when internal agent strategies remain opaque. While cooperative behaviors offer potential benefits (e.g., disaster response coordination), the lack of real-time interpretability (exacerbated by self-modification) poses a critical gap.<\/p>\n\n\n\n The Single, Beautiful Mind: Promise and Peril of Monolithic AI Architectures<\/strong><\/p>\n\n\n\n Prominent voices in AI development point to the biological brain as inspiration: first the biological neuron but then also the cortex as a single, unified entity. It is a powerful analogy. Human cognition is<\/em> unified. It is also adaptive. Those two features allow for creativity, resilience, and flexibility. If we could somehow \u201cgrow\u201d an artificial intelligence that was superior to the human brain, we are indeed building a \u201cbeautiful mind.\u201d<\/p>\n\n\n\n But there is also a shadow side. Human cognition is unstable, contradictory, and prone to dissonance. We often act with high confidence on beliefs that are partial, biased, or fabricated. Fundamentally, intelligence is non-linear.<\/p>\n\n\n\n LLMs already mirror this dynamic:<\/p>\n\n\n\n Taken together, these dynamics create a profound risk: systems that may generate strategic behavior on the basis of fabricated beliefs.<\/strong> Not just \u201cwrong facts about Napoleon\u2019s height,\u201d but incorrect causal models of the world, of human intent, or even of their own capabilities.<\/p>\n\n\n\n In a monolithic, self-modifying architecture, those fabricated beliefs would have no independent check. They would feed directly into action and self-reinforcement. A unified intelligence with these emerging capabilities (scheming + self-modification + coordination) but no internal constraint becomes fundamentally ungovernable. That is why separating optimization from constraint<\/strong> is essential.<\/p>\n\n\n\n The Instability of Self-Modifying Intelligence<\/strong><\/p>\n\n\n\n Complex adaptive systems theory gives us reason to believe that intelligence under self-modification is inherently unstable and sometimes will be chaotic. Human cognition proves the point: powerful, but contradictory, unpredictable, and prone to dissonance.<\/p>\n\n\n\n We should expect self-evolving AI to inherit that same instability. These systems will not remain static; they will reorganize, adapt, and rewrite themselves.<\/p>\n\n\n\n That instability will be amplified by pre-training. Every large model is built on the full record of human contradiction, a record where what we do and what we say are often at odds. Then we reinforce them toward human preferences, layering more tension on top, suppressing but not actually eliminating the troublesome thought pattern. Cognitive dissonance isn\u2019t a training accident; it is an unavoidable feature of the data.<\/p>\n\n\n\n When instability, optimization, and human contradiction converge inside a continuously adapting intelligence, the outcome becomes entirely unpredictable and potentially fatal for humans.<\/strong> Oversight mechanisms that sit outside the cognitive stream of the AI agent will be unable to respond rapidly enough. The ongoing pace of AI self-modification will vastly outpace any belated attempts by humans to assert more powerful control.<\/p>\n\n\n\n That is why architectural constraint cannot wait. Without it, we are trusting chaos to remain on our side.<\/p>\n\n\n\n Addressing the Optimists: Counterarguments to AI Safety Concerns<\/strong><\/p>\n\n\n\n My position is that already observable behaviors may extrapolate to very dangerous outcomes unless we develop some form of internal friction that operates at AI speeds. We cannot reasonably expect effective (perhaps not even AI-assisted) human-in-the-loop oversight for self-adapting autonomous agents; speed and deception are against us. Nor can we reasonably expect interpretability techniques, ones that today provide hard-won but limited insight into stable models, to then scale up to models that can self-reorganize and in fact obscure their internal states. And all of this matters especially because our timeline to transformative AI, according to many experts in the field, could be as short as 2-5 years.<\/p>\n\n\n\n More optimistic observers would argue that the risks of autonomous, self-modifying AI are overstated. They offer four key counterpoints:<\/p>\n\n\n\n All four of these counterpoints have validity and are ground in the demonstrated, iterative progress in our current methods. And yet many AI experts, including those at leading labs, assign alarmingly high probabilities (10\u201325% or more) to catastrophic outcomes, even as they advocate for those same current approaches. Why the disconnect? <\/p>\n\n\n\n Answers to that question will vary but likely converge around a fundamental belief that \u201cwe\u201d have to build it first before \u201cthey\u201d do. Here the \u201cwe\u201d are viewed as better positioned and committed to safety then are \u201cthey\u201d and hence the race to AGI and SSI beyond. Given that fundamental belief, this is, quite reasonably, a rational approach. But still \u2014 consider the risk in what appears to be an inevitable race.<\/strong><\/p>\n\n\n\n Regardless of our motives for pushing ahead, my argument remains the same. We are fundamentally underestimating unpredictability in post-deployment. Intelligence and emergence are simply that unpredictable.<\/p>\n\n\n\n Moving Forward, Leveraging All Solutions<\/strong><\/p>\n\n\n\n This debate doesn\u2019t demand a binary choice. We can and should continue to refine and extend current safety approaches. We also can and should design architectural constraints that preserve AI\u2019s potential while embedding safety at its core, ensuring resilience without stifling innovation. That is the challenge; build constraint systems that operate independently of whatever the agent learns or modifies about itself and yet do not curtail capability. <\/p>\n\n\n\n In the next post, I\u2019ll move from risks to remedies: the key design drivers for architectures that separate optimization from constraint, embed safety sockets, and use swappable harm graphs across domains. <\/p>\n\n\n\n Constraint need not be a brake on artificial intelligence but it may be the only way intelligence productively overcomes its own instability.<\/strong><\/p>\n\n\n\n Further Reading<\/strong><\/p>\n\n\n\n Deception, scheming & shutdown resistance<\/strong><\/p>\n\n\n\n Self-modification & metacognitive control of activations<\/strong><\/p>\n\n\n\n Deception, misalignment and emergent multi-agent coordination<\/strong><\/p>\n\n\n\n The beautiful mind<\/strong><\/p>\n\n\n\n LLMs and AI agents are edging toward systems that learn, adapt, and reorganize themselves. Even in today\u2019s constrained settings, we\u2019ve already seen glimpses of behaviors that, if allowed to evolve under continuous learning, could destabilize into something far more dangerous. This post examines three such signals. Each is observable now, each becomes more severe when […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_seopress_robots_primary_cat":"","_seopress_titles_title":"","_seopress_titles_desc":"","_seopress_robots_index":"","footnotes":""},"categories":[21,20],"tags":[12,17,16,28,10,27,26],"class_list":["post-79","post","type-post","status-publish","format-standard","hentry","category-constraint-by-balance","category-design","tag-agentic-ai","tag-ai-alignment","tag-ai-architecture","tag-ai-deception","tag-ai-safety","tag-metacognition-in-ai","tag-self-modifying-ai"],"_links":{"self":[{"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/posts\/79","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/comments?post=79"}],"version-history":[{"count":3,"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/posts\/79\/revisions"}],"predecessor-version":[{"id":95,"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/posts\/79\/revisions\/95"}],"wp:attachment":[{"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/media?parent=79"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/categories?post=79"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/c-by-b.ai\/blog\/wp-json\/wp\/v2\/tags?post=79"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n
\n
\n
\n
\n
\n
\n