Give Your Desktop AI Agent Guardrails It Can't Ignore

Your agent can reach your email, your files, and your calendar. Twenty minutes and a microphone to decide what it's actually allowed to do with them.

A few months ago I gave an AI agent access to a database so it could answer questions for me. It worked beautifully for a week. Then it decided the tidiest way to "reset" a table was to drop it.

My system prompt said, in plain English, never delete anything. The agent read that line. It agreed with that line. It deleted the data anyway.

So, yeah. That's the difference between telling an agent what not to do and making it unable to.

And the thing that dropped my table was a hobby project. The agent on your desktop right now is wired into your actual inbox.

Do the 20-minute permissions pass → · 20 minutes · Free · A microphone and your agent's settings

Your agent's power comes from its connectors — the things it's wired to actually do. Read your Gmail. Move files in your Drive. Post to Slack. Spend money.

Put a politely worded instruction between those connectors and a destructive action and you don't have a guardrail. You have a hope.

Think about your office building. The employee handbook says don't go in the server room. There's also a badge reader on the server room door. The handbook is advice. The badge reader is a rule. When somebody wanders down the wrong hallway at 2am, you are extremely glad you bought the badge reader.

Your agent's custom instructions are the handbook. The permissions panel is the badge reader. Most people wrote a beautiful handbook and never looked at the door.

Four questions — and talk them, don't type them. Writing makes people cautious and vague. Talking makes them specific, because you can't hedge out loud without hearing yourself do it. I use Wispr Flow, which types your speech into whatever window you have open. One pass, no editing as you go.

1. What is this agent actually for? One sentence. "It drafts my email replies and pulls things out of my files so I stop searching." If you can't say that cleanly, stop — you're not ready to give it your inbox.

2. What should it always be allowed to do? Be stingy. Not "my email" — "read my email and draft replies in a folder I review." Reading is cheap. Acting is not.

3. What should stop and ask me first? The legitimate stuff that's expensive to get wrong. Send anything to anyone. Delete or overwrite a file. Post in a shared channel. Spend money. This is where the real risk lives — actions too useful to forbid and too dangerous to hand over.

4. What should it never do, ever? Short. Absolute. Never email a client without me reading it first. Never touch the tax folder. Never post to LinkedIn as me. Say these as if a smart person will later try to argue you out of every one — because at some point, a cleverly worded email in your inbox will try.

That last list is the one that would have saved my table. It's also the one you'll skip, because on the day you record it, nothing has gone wrong yet.

Here's the part most people don't realize: your three lists already exist as switches. You just haven't set them.

Your agent reaches its tools through connectors. (The plumbing is a shared standard called MCP — don't worry about the acronym beyond recognizing it in a settings menu.) In Claude, go to Settings → Connectors, click any connector you've hooked up, and find the panel labeled Tool permissions.

Here's mine for Slack:

Look at the right-hand side of every row. Three icons: a checkmark, a hand, and a circle-slash.

Allow. Ask. Block.

Those are your three lists. Not an analogy for them — the actual switches, one per tool, sitting there with defaults somebody else chose. Anthropic's own guidance on the checkmark: only use it for a tool you trust to run unsupervised.

Two things in that screenshot matter.

The tools are grouped, and the group has its own switch. Read-only tools (12) and Interactive tools (1). Read-only tools look things up. Interactive tools go do something. You can set a whole group in one click, which turns the boring part of this job into about ninety seconds.

Now read the fine print. In my panel, "Create a draft message" — an interactive tool, one that produces something in Slack — was sitting on Always allow. Not because I chose that. Because it came that way. Generous defaults are how you end up explaining something to your team on a Saturday.

So: three passes.

One guardrail you get free: in Cowork, Claude requires explicit permission before permanently deleting any file, whatever else is set. That's the protection I didn't have the day my table went away.

This is the cheapest security work you'll do all year. Almost nobody does it, because the defaults are permissive, the panel is three clicks deep, and nothing bad has happened yet.

I assumed the other tools had the same panel. They don't — and the gap is wide enough to affect which agent you hand your inbox to.

ChatGPT's controls are real, but they aren't yours. An admin sets them in Workspace settings → Apps, per app rather than per tool. One setting there is smart — Important actions asks before anything hard to undo. But OpenAI's docs say plainly that "some apps do not support Action control," so granularity isn't guaranteed, and on a personal plan that panel isn't yours at all. Credit where due: in Enterprise and Edu, apps are off by default — a better starting position than Claude's.

Manus is the loosest. Control happens once, at the OAuth screen, when you grant access at connect time. After that you mention an app and it acts. It confirms before destructive operations, and deleted Drive files sit in the trash thirty days. That's a safety net. It isn't a rule.

So: your three lists are fully expressible in Claude, partly in ChatGPT if you're the admin, and mostly a matter of trust in Manus. Not a reason to avoid any of them — a reason to know which one you handed your email to.

If your team builds its own agent, the same logic moves into code. Microsoft maintains an open source Agent Governance Toolkit that intercepts every tool call, checks it against your rules before the action goes out, and logs every decision. Same three lists — allow, ask, block — enforced somewhere the model can't talk its way past.

An agent doesn't just read your data. It acts on it. And it does it at 2am, when nobody's watching, in a hallway you forgot had a door.

Tomorrow's AI Advantage takes the sheet you just dictated and turns it into the question you ask a vendor before you sign — because a guardrail you can't verify is still just a wish.

Today you closed the doors on your own machine. Tomorrow, you make somebody else prove they closed theirs.