Build an AI agent rollback plan by classifying every action as retryable, reversible, compensable or irreversible. Give each run a stable operation ID, record the state before and after every consequential tool call, and define the recovery action before launch. A failed run closes only when the business system reaches a verified valid state.
This matters once an agent can change a CRM record, send a message, issue a refund or move work between systems. A timeout can leave the action completed at the destination while the agent records failure. Retrying blindly then creates a second consequence. Recovery design prevents one uncertain tool result from becoming duplicated work, financial loss or a customer-facing mistake.
Start with the business state you need to protect
Write the valid end state before designing retries. For an invoice workflow, that state may be one approved payment matched to one invoice with an audit reference. For customer success, it may be one reviewed renewal message recorded against the correct account. The agent’s final response is evidence about the run; the system of record determines whether recovery succeeded.
Map each step from trigger to outcome and mark every external state change. Reads still matter when stale or restricted evidence changes the decision, but rollback work concentrates on writes, messages, approvals and commitments.
Use a table like this:
| Action | Failure exposure | Recovery class | Evidence required | Owner |
|---|---|---|---|---|
| Read account context | stale or unauthorised input | refresh and rerun | source version and permission basis | data owner |
| Draft customer email | weak or incorrect content | regenerate or reject | draft version and review decision | account owner |
| Send customer email | wrong recipient or commitment | irreversible; remediate | exact message, recipient and send receipt | commercial owner |
| Update CRM stage | wrong or duplicate state | reverse or compensate | before/after values and operation ID | RevOps |
| Trigger refund | duplicate financial action | idempotent retry, then reconcile | payment reference and processor state | finance owner |
The table forces a commercial decision at each tool boundary. Engineering can then implement the required state capture and controls without inventing policy during an incident.
Give the whole run one operation ID
Generate a stable operation ID from the originating business event. Preserve it through model calls, tool calls, approvals, retries, handoffs and compensation. Derive a separate stable idempotency key for each destination action. A retry keeps that child key; a different action receives its own key, linked to the same parent operation.
The operation ID joins three records that otherwise drift apart:
- what the agent intended to do;
- what each tool attempted;
- what the destination system confirms happened.
A model run ID alone is too narrow when one run touches several services. A CRM update may succeed before a messaging API times out. Recovery needs the parent business operation and the status of each child action.
Store an action ledger under that ID. For every consequential step, capture the tool and arguments, initiating identity, authority used, pre-action state, attempt number, response, observed post-action state and available compensation. Apply access and retention controls because this record can contain sensitive evidence and customer data.
Separate safe retries from unknown outcomes
Classify failures before retrying:
- Confirmed rejection: the destination proves that no state changed. A corrected retry can proceed within policy.
- Transient failure before execution: connection or validation failed before the destination accepted the request. Retry with bounded backoff.
- Unknown outcome: the request left the system, but the response was lost or timed out. Reconcile against the destination before another attempt.
- Confirmed partial completion: one or more prior steps changed state. Continue, compensate or escalate according to the recovery plan.
- Policy rejection: authority, permission or approval was denied. Close or escalate; technical retry will not repair the policy condition.
Temporal’s documentation on activity failures separates scheduling, execution and overall timeouts because each failure says something different about whether work started. The same discipline applies even when the workflow uses another runtime. A timeout label needs enough precision to tell the recovery service whether another attempt is safe.
For endpoints that support it, idempotency lets the destination recognise a repeated logical action. Use a stable child-action key and follow that endpoint’s retention and parameter rules. Pair this with reconciliation; an idempotency key cannot repair a wrong but successfully executed action.
Check the provider’s deduplication window before replaying an old run
A saved idempotency key does not establish how long the destination will recognise it. Stripe’s idempotency documentation says keys can be removed after they are at least 24 hours old. Reusing a key after removal generates a new request. Stripe also compares request parameters and returns an error when a reused key carries different parameters.
That changes the recovery decision. An operator reopening an incident after a weekend must establish what happened at the destination before replaying a write. The local action ledger can retain the key long after the provider has stopped recognising it. Other APIs have their own contracts; check the actual endpoint rather than adopting Stripe’s window as a general rule.
Add these fields to each action’s recovery record:
| Record | Decision it supports |
|---|---|
| Parent event and stable child-action key | Distinguish a retry from another authorised action in the same workflow |
| Exact original parameters and destination reference | Find the original effect and prevent a changed payload being treated as a replay |
| First-attempt time and documented key-retention rule | Identify late recovery that can fall outside provider deduplication |
| Authoritative lookup and its consistency limits | Decide whether the action occurred, rather than interpreting a timeout as rejection |
| Escalation owner when the outcome stays unknown | Keep an unresolved write from becoming an automatic second attempt |
For an illustrative CRM workflow, separate creating a task from changing an opportunity stage. They share the business event but require different child-action identities. If task creation succeeds and the response disappears, search for the task using a destination-supported reference. An empty result from a delayed index is insufficient evidence that creation failed. Hold the replay until an authoritative check resolves the outcome or a named owner chooses the remediation.
Stripe returns the first saved response for the same key even when it is a 500 error. Switching keys just to get past that response can authorise another execution. Preserve the uncertain outcome and follow the provider’s reconciliation procedure before authorising another write.
Rehearse late recovery as well as immediate retries
Use a sandbox or controlled test double to exercise the destination’s retention contract. These are acceptance tests to run, not reported results:
- Lose the response after a successful write, then retry within the documented window. Verify one business effect and the original destination reference.
- Change the payload while retaining the key. Confirm the provider’s documented rejection behaviour and prevent the workflow from silently issuing a replacement key.
- Model an expired or removed key with an already-completed action. Require reconciliation to find that action and block duplicate execution.
- Make the lookup unavailable or inconclusive. Verify that the run remains unresolved, with downstream actions blocked and an owner assigned.
Include a successful new-action control to check that deduplication has not blocked legitimate work. Keep destination records with the test evidence; a log line saying “duplicate prevented” does not prove that only one task or payment exists.
Design compensation around business rules
Rollback language suggests that the system can restore an earlier snapshot. Distributed workflows rarely offer that clean guarantee. Another user may have changed the record, a payment may have settled or a customer may have acted on a message.
Microsoft’s Compensating Transaction pattern treats recovery as a second, application-specific workflow. The system records completed steps and the information required to undo or offset them. Compensation can fail too, so each recovery step needs progress tracking, idempotency and an escalation path.
For each consequential action, define:
- the condition that triggers compensation;
- the command that creates a valid state;
- the evidence that command needs;
- whether human approval is mandatory;
- the order relative to other recovery steps;
- the terminal state and verification query.
A CRM stage change may be reversed after checking that no colleague has progressed the opportunity. A refund cannot be “unpaid” through a database edit; the business may need a new charge, customer consent and finance review. Recovery follows the domain’s rules, not the inverse of the original API call.
Put points of no return late in the workflow. Complete source checks, policy validation and approval before sending an external message or making a binding financial change. This reduces the number of incidents that require remediation rather than technical compensation.
Make approval a resumable state
Approval should bind a reviewer to the exact action parameters they saw. Store the artefact version, proposed tool call, authority basis, decision, timestamp and expiry. If the action changes after approval, request a new decision.
OpenAI’s guardrails and human review documentation records an approval interruption and resumes the same run from stored state. It also places validation beside the tool that creates the side effect, because agent-level input and output checks do not cover every nested call.
Microsoft Agent Framework uses a similar request-and-response model for human-in-the-loop workflows. Pending requests can persist in checkpoints and reappear when the workflow resumes. That behaviour needs an expiry rule: an approval granted against yesterday’s price, account status or evidence set may be invalid today.
Set a response deadline and define what happens when nobody answers. Consequential actions should pause or close safely. The run must never interpret missing approval as permission.
Decide when to continue, compensate or stop
Recovery has three routes:
- Continue forward when the destination confirms prior steps and the remaining path stays valid.
- Compensate when completed actions can be offset under known business rules.
- Stop for reconciliation when the state is uncertain, the action is irreversible or compensation requires judgement.
Choose the route with a recovery matrix rather than another model judgement made inside the failing run.
| Current state | Consequence | Default route |
|---|---|---|
| no external write occurred | low | bounded retry |
| write succeeded and later step failed | reversible | compensate, verify, then close |
| destination state is unknown | any material consequence | query system of record; block duplicate execution |
| irreversible action completed | customer, legal or financial | freeze downstream work and assign remediation |
| permission or approval failed | protected action | deny and record; require a new authorised request |
The matrix keeps recovery authority narrower than execution authority. An agent allowed to update pipeline data should not automatically gain permission to reverse contracts, contact customers or initiate financial correction.
Test the recovery path before production
Use representative cases from the intended workflow. Run each one far enough to create a controlled partial failure, then verify that the system reaches the declared valid state.
Test at least these conditions:
- The tool rejects the request before execution.
- The destination applies the action but the response times out.
- The same event arrives twice.
- A later step fails after an earlier write succeeds.
- Approval expires before the run resumes.
- Permission is revoked between planning and execution.
- The compensating action fails on its first attempt.
- An irreversible action requires manual remediation.
Connect each result to the evaluation suite and incident process. The AI agent troubleshooting guide helps locate the first divergence across context, judgement, tools, authority, handoffs and outcomes. Once diagnosed, preserve the case as a regression test with a verified destination state.
The AI agent pilot success criteria guide adds the release decision. A pilot should not scale when accepted outputs depend on recovery work the team has neither measured nor staffed.
Use a one-page rollback plan
Keep the operational record compact enough for product, engineering and the business owner to use during an incident:
- Workflow and owner: named business process, technical owner and recovery owner.
- Valid end state: the system-of-record condition that closes the operation.
- Operation identity: parent ID plus child-action idempotency keys.
- Action ledger: pre-state, request, response, post-state and authority evidence.
- Failure classes: confirmed rejection, pre-execution failure, unknown outcome, partial completion and policy denial.
- Recovery map: retry, continue, compensate, reconcile or remediate for each action.
- Approval rules: reviewer, exact payload, expiry and unavailable-reviewer behaviour.
- Points of no return: irreversible actions and the validations that precede them.
- Stop control: how new runs and downstream actions are suspended.
- Verification: destination query, reviewer decision and audit record required to close recovery.
This plan exposes implementation work before the agent receives broad authority. It also reveals where conventional automation should own transport, deduplication, state checks and compensation while model judgement stays inside the variable decision. The agent-versus-automation framework gives a step-level method for drawing that boundary.
Recovery ends with a verified business state
A kill switch can stop the next action while leaving previous changes unresolved. A successful compensation call can still produce the wrong account balance. Close the incident only after the authoritative system confirms a valid state, a named owner accepts the result and remaining customer or financial consequences have a recorded route.
Model Operator is a Dubai-based AI product studio led by Alexander Chrisostomou. Through AI Initiative Consulting, Alexander maps the workflow, recovery responsibilities and acceptance conditions before agreeing the implementation scope. Bring one consequential workflow and a failure you cannot safely replay to start a build conversation.