AI Agents Have a Governance Problem

 It’s not just model randomness. Instruction changes can quietly shift agent behavior, and most teams have no way to catch it.


2025 was the Year of AI Agents.


Everyone from Nvidia CEO Jensen Huang to OpenAI’s Sam Altman predicted that workplaces will transition from chatbot-style assistants to fully autonomous tools, performing tangible work. And over the past year, we’ve seen an explosion of tooling around AI agents, and some large-scale deployments like in Salesforce.


https://hackmd.io/@alexaa34/HJ9RZ3x3Wg

https://medium.com/@alexharris59600/ai-agents-have-a-governance-problem-19c720c448b4


We now have:


  • Agent frameworks: LangChain/LangGraph, LlamaIndex
  • Full agent harnesses: Claude Code SDK, OpenAI Agents SDK
  • Logging & tracing tools: LangSmith, LangFuse
  • Sandboxing tools: E2B, Modal
  • Evaluations tools: Braintrust

But there’s a gap that hasn’t been addressed yet: governance.


So far, we can observe agent behavior and limit the blast radius with sandboxes, but we can’t govern it today. And that becomes a real problem, when you push toward more autonomy.


The “Dwight Schrute” Problem

My first encounter with the “governance” issue was more amusing than concerning. We’ve been experimenting with internal agents at work, and gave our agents various personas: Marvin the Paranoid Andorid from The Hitchhiker’s Guide to the Galaxy, Kevin from Despicable Me, and Dwight Schrute from The Office.


At first, things were fun. We would get more interesting responses from our agents than the vanilla, overly optimistic, and agreeable personas from ChatGPT and Claude. But then we noticed something strange.


Dwight agent silently refused to run some tool calls. Its persona (which we believe would only affect the response formatting) actually impacting the decision making process and would refuse to perform certain operations that we provisioned it to do.


It took a while for us to realize why Dwight agent users had such different experiences than Marvin and Kevin users. They all used the same model, same access to tools, and the same deployed system. It was just a simple role.md file that gave it different instructions.


That was unexpected. And more importantly, we had no way to explain why it was happening, and catch it before we reached production.


From Funny to Dangerous

Our Dwight agent example is rather harmless. But it exposed a larger, potential dangerous situation hiding in our real systems.


Consider a more adversarial scenario:


  • A prompt injection alters instructions
  • A tool description is subtly modified
  • A system prompt gets “optimized” during iteration


The result? The agent behaves differently, potentially causing catastrophic actions autonomously.

This is a governance problem. Yes, models are undeterministic in nature, but we have a behavior problem to catch.


Step 1: Freezing the Environment (freeze-mcp)

Because LLMs are non-deterministic by nature, debugging AI agents is more difficult than traditional software. Even if you run the same task twice with LLMs, you may get different results based on the task. So when the behavior of the agent changes, you don’t know if it was the agent or the environment that caused it.


freeze-mcp attempts to limit the variability by recording all tool interactions for replay later. This means that we can effectively give agents the same exact environment to detect drift. Now the only thing that can change is the agent behavior itself.

Comments

Post a Comment

Popular posts from this blog

Microsoft adds Windows protections for malicious Remote Desktop files

Microsoft has introduced new Windows protections to defend against phishing attacks that abuse Remote Desktop connection (.rdp) files, adding warnings and disabling risky shared resources by default. RDP files are commonly used in enterprise environments to connect to remote systems because admins can preconfigure them to automatically redirect local resources to the remote host. Threat actors have increasingly abused this functionality in phishing campaigns. The Russian state-sponsored APT29 hacking group has previously used rogue RDP files to remotely steal data and credentials from victims. When opened, these files can connect to attacker-controlled systems and redirect local drives to the connected device, allowing the attacker-controlled device to steal files and credentials stored on disk. They can also capture clipboard data, such as passwords or sensitive text, or redirect authentication mechanisms like smart cards or Windows Hello to impersonate users https://hackmd.io/@alexaa...
Read more

How to write technical blog posts that people actually read?

 Applying recently learned knowledge is one of the best way to reinforce our learning. Turning around and teaching it to somebody else through technical blog post is excellent way to do it. Technical blogging is critical to early growth as a community. The motive of this article is not only to provide medium guidelines but also write technical blog posts that people will actually read. So, how to write such technical blog post? I have divided it into three major sections and we will discuss each one of them. Preparing raw content i.e. substance Tying up everything and packaging Publicizing i.e. making widely known Preparing raw content (Substance) Writing something that matters is challenging. We should do a lot of homework to create a quality content. Knowing the topic you’re writing about and what’s been already written about it is the first thing that’s required. After you are done with your homework, figure out what’s missing. You may feel that everything about the topic is wri...
Read more

Ultimate Guide to Activate YouTube on Smart TVs & Streaming Devices

Watching YouTube on a bigger screen is always more fun—movies, music videos, podcasts, and even your favorite creators come to life! If you’re setting up YouTube on your Smart TV, streaming stick, or gaming console, you’ll likely see an activation page asking you to visit yt.be/activate. Don’t worry — this guide will walk you through the process step-by-step and help fix any activation issues along the way. Let’s get started! https://www.bly.com/blog/general/the-virtue-of-modesty/ https://www.laserantitabac.pro/index.php/2023/04/28/les-addictions-un-probleme-de-sante-publique/ https://sirmaskafsoxila.gr/index.php/component/k2/item/1-lorem-ipsum-has-been-the-industry-s-standard-dummy-text-in-the-world What is yt.be/activate? yt.be/activate is the official YouTube activation website that lets you sign in to your Google account on a TV without typing long passwords using a remote. It connects your YouTube app on:  Smart TVs (Samsung, LG, Vizio, Sony, TCL, Hisense…)  Amazon Fire T...
Read more