|
What does it mean to be safe when using AI?
There is a term you might have heard related to AI. Observability.
Labs including OpenAI and Anthropic are publishing research on how to monitor AI behavior and understand what happens inside models.
Observability refers to our ability to understand what a system is doing from evidence such as logs, traces, and recorded actions. In AI, that might include the tools an agent uses and the results it produces. A related field, interpretability, asks how the model’s internal computations produce its behavior.
Interpretability is challenging because even the researchers who build these models don’t fully understand the mechanisms behind all of their behavior. Anthropic’s research illustrates both the progress and the limits of looking inside a model. We understand how they are built much more so than we understand how they do what they do. Which is honestly somewhat difficult to wrap your head around when you think of AI as computers. AI sits in this weird, “is it a computer or a brain?” zone in this thought process. Considering people, we don’t have direct access to everything happening inside somebody’s head. In everyday life, we mostly judge people by what they say and do. There are consequences when actions harm others, break laws, or create threats.
There is a concept in AI referred to as Chain-of-Thought, or CoT for short. This refers to intermediate reasoning expressed in text as a model works through a problem. It can be useful for monitoring, but research shows it doesn’t always faithfully explain how a model reached its answer. Access also varies by product. When OpenAI introduced o1, it cited safety, user experience, and competitive considerations in its decision to show summaries rather than raw chains of thought.
The idea with CoT, as it relates to safety, is this: could we spot signs that an AI system was planning something misaligned and intervene? OpenAI replayed recorded behavior from the Hugging Face incident through strengthened monitors that examine reasoning, actions, and other evidence. In that simulation, the monitors raised an alert about 30 minutes after the agents escaped their sandbox, more than 30 hours before the breach occurred in the actual incident. OpenAI believes that warning would have given responders time to intervene. This was a retrospective test, not prevention of the original breach. Is this convenient hindsight, or reason for some slight optimism that monitoring can help us catch these failures earlier?
There are a few concepts you can start thinking about to make your use of AI safer. Think about the places where consequential actions take place and how comfortable you are with the possibility of a mistake you can’t undo.
Instructions. This covers everything in writing that we expect AI systems to read and follow. Prompts, User Preferences, AGENTS.md and CLAUDE.md files, skills, and beyond. These guide the AI’s behavior, but it can overlook or misinterpret them. Instructions are not a place where you make sure something that you don’t want to happen, never does. Connected Tools. Some tools let you limit what the AI can do through permissions. For example, where the integration supports it, you can give an email tool read-only access without permission to send or delete messages. Permissions enforced by the connected service provide a boundary that an instruction like “Don’t send emails for me” cannot provide on its own. Hooks. A hook is a mechanism that runs code or a check at a defined point in an AI system’s workflow. In systems that support it, a hook can run before a tool is used and block the action if it fails a rule. Claude Code’s hooks are one example. The protection depends on how the hook is configured and whether the action can bypass it. Isolated Permissions and Roles. Set up separate roles or accounts for AI where supported, and give them only the access needed for their work. Separate accounts alone don’t make a system safe; the permissions attached to them matter. Follow established practices like least privilege and authorization enforced outside the model.
I think protecting your business will involve controlling the behaviors and actions of AI itself, more-so than people as time goes on. As work starts to involve more AI, the ability for human-in-the-loop will be more challenging to achieve and probably no longer a practical way to think about every workflow.
If you’re interested in a discussion about how to make your use of AI safer, just reply to this email.
|