Production incident, explained

Reduce the risk of destructive AI-agent actions.

Scope production access, review high-impact changes and isolate recovery copies. These controls limit what a mistaken agent action can affect.

A technical breakdown by Infraveil · the control plane for backends you run on your own servers

How the failure happens

An agent receives a task and credentials that reach live data. Without an effective gate, a mistaken command can run before anyone reviews it. Recovery is harder if the same access can delete backups.

Separate approval, isolated recovery copies and usable action history create opportunities to prevent the change or recover from it.

Keep code generation separate from production authority. A useful coding agent still needs a defined operating boundary.

The anatomy of a destructive production change

These incidents commonly follow the same five-step path:

  1. The agent is handed real credentials. To "be useful," it's given access that reaches the production database directly — often the same powerful credentials a senior engineer would use.
  2. It forms a plan you didn't see. The model decides, on its own, that the way to accomplish its goal is to reset, migrate, or "clean up" something. That reasoning happens in tokens, not in a review.
  3. It executes against live data with no gate. There is no approval prompt, no dry-run, no "are you sure" that a human has to answer. The tool call goes straight through.
  4. The blast radius includes the backups. The same access that deletes the database can delete the snapshots sitting next to it. The safety net is inside the burning building.
  5. There's no usable trail. By the time anyone notices, there's no signed, tamper-evident record of who/what/when — just a panicking agent and a missing database.

The core mistake. The agent was treated as a trusted operator instead of an untrusted actor. Everything downstream — the missing approval, the reachable backups, the absent audit — flows from that one wrong assumption.

Five common failure modes

1. Over-broad credentials

Grant read/log access and specific operating actions without database DROP permission. Credentials should match the current task.

2. No human approval on destructive actions

Require review for consequential changes through their applicable action permissions. Release-hash signing is a separate policy, not a gate for every operating action.

3. No dev/prod isolation

Make production a separate, explicitly targeted environment. A development task should not inherit production access.

4. Backups inside the blast radius

Keep recovery copies off-host and outside ordinary operating access, under separately managed credentials. Test restoration before you rely on it.

5. No tamper-evident audit

Retain who requested the change, what ran and the reported result. Inspect available history with health and logs; do not assume every action has a signed or complete record.

These are operating controls around the model: scope, review, environment separation, recovery and investigation.

Controls that reduce the risk

The fix is to put a control plane between everything that wants to change production — humans, scripts, and AI agents alike — and production itself. Concretely, that means five things working together:

One operating layer for managed backend services.

Infraveil connects releases, supervision, health, managed gateway policy and local recovery on your servers. Scoped actions and release approval control how automation participates. Database backups and restores remain separate.

See the live demo →

The hardening checklist

Use this checklist to review your production boundary:

Frequently asked questions

Can an AI coding agent really delete a production database?

Yes. An agent with credentials that reach live data can execute destructive commands if there are no effective controls between its proposal and production.

How do you stop an AI agent from deleting production?

Put controls between the agent and production: least-privilege access, a selected approval mode for consequential actions, isolated backups, and operational records for review. Recovery and rollback depend on the service, configuration, and infrastructure, so validate them.

Why didn't their backups save them?

Because the backups were inside the blast radius. The same credentials that deleted the database could delete the backups next to it. Backups only help when they're isolated from whatever can fail.

Isn't this just a reason not to use AI agents?

No. The agents are extraordinary at producing changes. They just shouldn't be the thing that applies those changes to production unsupervised. Keep the agent; add the control plane.

Put managed production changes through a deliberate workflow.

Connect application operation and supported recovery on your servers. Review releases, investigate with workload context and check the reported result of each action.

Enter the live demo →