AI agents need intern-level oversight to prevent runaway risks, experts warn

At a glance:

• AI agents require strict oversight and specific instructions to avoid unintended actions, experts say.
• Context, intent, and restraint are critical for managing agent behavior and preventing shadow AI risks.
• Traditional software development practices are inadequate for governing non-deterministic AI systems.

The intern analogy: why AI agents need close supervision

AI agents are rapidly evolving from basic chatbots into autonomous digital workers capable of executing complex tasks across applications and data. However, this leap in capability introduces significant security and governance challenges, according to a panel of experts at the Snowflake Summit in San Francisco. Mayank Agarwal, founder and CTO of Resolve AI, emphasized that agents without proper constraints can cause serious issues, stating, "You may tell the agent to buy you shoes, and before you know it, it has bought you a car."

The comparison to human interns is apt: just as interns require clear guidance and oversight, AI agents demand precise instructions and continuous monitoring. Nancy Wang, CTO of 1Password, highlighted the importance of understanding an agent's authority and data access scope. "It's not just enough to know what this agent was created to do. You also have to know things like whose authority it is acting under and what it's going to do, for example, with data it's accessing," she said. This shift in mindset is essential as agents gain more autonomy.

Restraint, context, and intent: the pillars of agent governance

The panel identified restraint, context, and intent as the core principles for managing AI agents. Agarwal stressed that developers must carefully consider the permissions granted to agents, as their unpredictable behavior can lead to unforeseen consequences. "You have to think very hard about what permissions you're giving the agent. You can't just expect an agent to stay on the straight and narrow. You have to put these ironclad constraints around it to limit what it's able to do," he explained.

Unlike traditional software development, where API interactions are predictable and linear, agentic systems operate dynamically. Agarwal noted, "If you go back just two years, an engineer knew exactly how they were going to connect APIs across different systems. The whole thing was very predictable: A is going to call API B, B is going to do this with that data, and call C, and do this with that data. In the agentic world, it's completely unpredictable."

Shadow AI and the challenges of visibility

The rise of shadow AI—unauthorized or unmonitored agent deployments—poses a significant risk. Jason Merrick, SVP of product at Tenable, shared a cautionary example: "We had a client that had 12 OpenClaw instances within their framework, with access to API feeds, source code, and a contractor using Telegram to communicate. What could go wrong, right?"

These hidden agents complicate accountability, as their actions can blur the lines between human and automated behavior. Wang pointed out the ambiguity: "Who actually took an action against this system? Is it a human? Is it a service account? Or is it an agent? Your team probably doesn't know, or there's not 100% certainty to that answer."

Balancing productivity and control in agent deployment

While governance is crucial, experts caution against overly restrictive measures that stifle innovation. Wang advised, "You don't want to just block everything or firewall everything."

Merrick emphasized proactive monitoring: "Look at the user pieces the employees are creating -- through Copilot, Claude Chat, or Gemini. Look at their configurations. Is AI misconfigured? What type of data is it accessing? And be able to take action on that."

Designing guardrails for non-deterministic agents

Traditional identity and access management practices are insufficient for AI agents, which Wang described as "non-deterministic beings." She advocated for a hybrid approach: "You want predictable controls, but also, you don't want to constrain them so much that it no longer gets you productivity gains."

The key takeaway is the need for explicit intent and continuous oversight. Wang concluded, "Sometimes they still veer off the desired path. Whether you think about governing agents or whether you think about full agent traces comes back to full visibility, remediation, and making sure that you set the right intent from the get-go -- and that intent must persist across every step, every action that the agent takes."