Build an AI agent rollback plan by classifying every action as retryable, reversible, compensable or irreversible. Give each run a stable operation ID, record the state before and after every consequential tool call, and define the recovery action before launch. A failed run closes only when the business system reaches a verified valid state.
This matters once an agent can change a CRM record, send a message, issue a refund or move work between systems. A timeout can leave the action completed at the destination while the agent records failure. Retrying blindly then creates a second consequence. Recovery design prevents one uncertain tool result from becoming duplicated work, financial loss or a customer-facing mistake.
Start with the business state you need to protect
Write the valid end state before designing retries. For an invoice workflow, that state may be one approved payment matched to one invoice with an audit reference. For customer success, it may be one reviewed renewal message recorded against the correct account. The agent’s final response is evidence about the run; the system of record determines whether recovery succeeded.
Map each step from trigger to outcome and mark every external state change. Reads still matter when stale or restricted evidence changes the decision, but rollback work concentrates on writes, messages, approvals and commitments.
Use a table like this:
| Action | Failure exposure | Recovery class | Evidence required | Owner |
|---|---|---|---|---|
| Read account context | stale or unauthorised input | refresh and rerun | source version and permission basis | data owner |
| Draft customer email | weak or incorrect content | regenerate or reject | draft version and review decision | account owner |
| Send customer email | wrong recipient or commitment | irreversible; remediate | exact message, recipient and send receipt | commercial owner |
| Update CRM stage | wrong or duplicate state | reverse or compensate | before/after values and operation ID | RevOps |
| Trigger refund | duplicate financial action | idempotent retry, then reconcile | payment reference and processor state | finance owner |
The table forces a commercial decision at each tool boundary. Engineering can then implement the required state capture and controls without inventing policy during an incident.
Give the whole run one operation ID
Generate a stable operation ID from the originating business event. Preserve it through model calls, tool calls, approvals, retries, handoffs and compensation. Every destination that supports idempotency should receive that identifier.
The operation ID joins three records that otherwise drift apart:
- what the agent intended to do;
- what each tool attempted;
- what the destination system confirms happened.
A model run ID alone is too narrow when one run touches several services. A CRM update may succeed before a messaging API times out. Recovery needs the parent business operation and the status of each child action.
Store an action ledger under that ID. For every consequential step, capture the tool and arguments, initiating identity, authority used, pre-action state, attempt number, response, observed post-action state and available compensation. Apply access and retention controls because this record can contain sensitive evidence and customer data.
Separate safe retries from unknown outcomes
Classify failures before retrying:
- Confirmed rejection: the destination proves that no state changed. A corrected retry can proceed within policy.
- Transient failure before execution: connection or validation failed before the destination accepted the request. Retry with bounded backoff.
- Unknown outcome: the request left the system, but the response was lost or timed out. Reconcile against the destination before another attempt.
- Confirmed partial completion: one or more prior steps changed state. Continue, compensate or escalate according to the recovery plan.
- Policy rejection: authority, permission or approval was denied. Close or escalate; technical retry will not repair the policy condition.
Temporal’s documentation on activity failures separates scheduling, execution and overall timeouts because each failure says something different about whether work started. The same discipline applies even when the workflow uses another runtime. A timeout label needs enough precision to tell the recovery service whether another attempt is safe.
Idempotency protects the destination when the same logical action arrives twice. Use the operation ID, or a stable child-action key, so repeated requests return the original result rather than applying the side effect again. Pair this with reconciliation; an idempotency key cannot repair a wrong but successfully executed action.
Design compensation around business rules
Rollback language suggests that the system can restore an earlier snapshot. Distributed workflows rarely offer that clean guarantee. Another user may have changed the record, a payment may have settled or a customer may have acted on a message.
Microsoft’s Compensating Transaction pattern treats recovery as a second, application-specific workflow. The system records completed steps and the information required to undo or offset them. Compensation can fail too, so each recovery step needs progress tracking, idempotency and an escalation path.
For each consequential action, define:
- the condition that triggers compensation;
- the command that creates a valid state;
- the evidence that command needs;
- whether human approval is mandatory;
- the order relative to other recovery steps;
- the terminal state and verification query.
A CRM stage change may be reversed after checking that no colleague has progressed the opportunity. A refund cannot be “unpaid” through a database edit; the business may need a new charge, customer consent and finance review. Recovery follows the domain’s rules, not the inverse of the original API call.
Put points of no return late in the workflow. Complete source checks, policy validation and approval before sending an external message or making a binding financial change. This reduces the number of incidents that require remediation rather than technical compensation.
Make approval a resumable state
Approval should bind a reviewer to the exact action parameters they saw. Store the artefact version, proposed tool call, authority basis, decision, timestamp and expiry. If the action changes after approval, request a new decision.
OpenAI’s guardrails and human review documentation records an approval interruption and resumes the same run from stored state. It also places validation beside the tool that creates the side effect, because agent-level input and output checks do not cover every nested call.
Microsoft Agent Framework uses a similar request-and-response model for human-in-the-loop workflows. Pending requests can persist in checkpoints and reappear when the workflow resumes. That behaviour needs an expiry rule: an approval granted against yesterday’s price, account status or evidence set may be invalid today.
Set a response deadline and define what happens when nobody answers. Consequential actions should pause or close safely. The run must never interpret missing approval as permission.
Decide when to continue, compensate or stop
Recovery has three routes:
- Continue forward when the destination confirms prior steps and the remaining path stays valid.
- Compensate when completed actions can be offset under known business rules.
- Stop for reconciliation when the state is uncertain, the action is irreversible or compensation requires judgement.
Choose the route with a recovery matrix rather than another model judgement made inside the failing run.
| Current state | Consequence | Default route |
|---|---|---|
| no external write occurred | low | bounded retry |
| write succeeded and later step failed | reversible | compensate, verify, then close |
| destination state is unknown | any material consequence | query system of record; block duplicate execution |
| irreversible action completed | customer, legal or financial | freeze downstream work and assign remediation |
| permission or approval failed | protected action | deny and record; require a new authorised request |
The matrix keeps recovery authority narrower than execution authority. An agent allowed to update pipeline data should not automatically gain permission to reverse contracts, contact customers or initiate financial correction.
Test the recovery path before production
Use representative cases from the intended workflow. Run each one far enough to create a controlled partial failure, then verify that the system reaches the declared valid state.
Test at least these conditions:
- The tool rejects the request before execution.
- The destination applies the action but the response times out.
- The same event arrives twice.
- A later step fails after an earlier write succeeds.
- Approval expires before the run resumes.
- Permission is revoked between planning and execution.
- The compensating action fails on its first attempt.
- An irreversible action requires manual remediation.
Connect each result to the evaluation suite and incident process. The AI agent troubleshooting guide helps locate the first divergence across context, judgement, tools, authority, handoffs and outcomes. Once diagnosed, preserve the case as a regression test with a verified destination state.
The AI agent pilot success criteria guide adds the release decision. A pilot should not scale when accepted outputs depend on recovery work the team has neither measured nor staffed.
Use a one-page rollback plan
Keep the operational record compact enough for product, engineering and the business owner to use during an incident:
- Workflow and owner: named business process, technical owner and recovery owner.
- Valid end state: the system-of-record condition that closes the operation.
- Operation identity: parent ID plus child-action idempotency keys.
- Action ledger: pre-state, request, response, post-state and authority evidence.
- Failure classes: confirmed rejection, pre-execution failure, unknown outcome, partial completion and policy denial.
- Recovery map: retry, continue, compensate, reconcile or remediate for each action.
- Approval rules: reviewer, exact payload, expiry and unavailable-reviewer behaviour.
- Points of no return: irreversible actions and the validations that precede them.
- Stop control: how new runs and downstream actions are suspended.
- Verification: destination query, reviewer decision and audit record required to close recovery.
This plan exposes implementation work before the agent receives broad authority. It also reveals where conventional automation should own transport, deduplication, state checks and compensation while model judgement stays inside the variable decision. The agent-versus-automation framework gives a step-level method for drawing that boundary.
Recovery ends with a verified business state
A kill switch can stop the next action while leaving previous changes unresolved. A successful compensation call can still produce the wrong account balance. Close the incident only after the authoritative system confirms a valid state, a named owner accepts the result and remaining customer or financial consequences have a recorded route.
Model Operator is implementing and validating governed run, approval, audit and truth contracts for company workflows. Through AI Initiative Consulting or an Agentic Company Brain build, the work starts with one consequential workflow and maps its action ledger, recovery classes, approval points and verification state. Bring the workflow and one failure you cannot safely replay to alexander@modeloperator.io or start a conversation.