Idempotent AI Automation: How to Stop Duplicates, Retries and Replays from Doing Damage
By Abhishek Patoliya · 2026-09-30
Here is a failure that almost every automation builder meets eventually. A workflow creates a customer record and sends a welcome email. The email service is slow, the request times out, and the workflow's retry logic dutifully tries again. The first request had actually succeeded. The customer now has two records and two emails, and the sales team wonders why the new system "sends spam."
Nothing in that story was a bug in the ordinary sense. Every component did what it was told. The problem is that the workflow was not designed to be idempotent: safe to run more than once for the same input without changing the end result beyond the first run.
Idempotency is the single most useful reliability idea in automation, and it matters even more when a language model is involved, because model steps are slow, occasionally fail, and get retried a lot. This article explains what idempotency means in practice, where duplicate work comes from, how to design duplicate-safe workflows with concrete patterns and code, and how to test that they really are. It draws on primary documentation from MDN and Stripe, and on n8n's webhook behaviour.
What idempotent actually means
The formal definition comes from HTTP. MDN's reference describes an idempotent method as one where making the same request multiple times has the same effect on the server as making it once. It lists which HTTP methods qualify:
GET, HEAD, OPTIONS, TRACE: safe (they do not modify state) and idempotent.
PUT and DELETE: not safe, but idempotent. Replacing a resource with the same content twice leaves the same result; deleting something twice leaves it deleted.
POST and PATCH: neither safe nor idempotent by definition. Posting the same order twice may create two orders.
Two points follow.
First, idempotency is about the resulting state, not about the response. The second DELETE may return "not found" instead of "deleted," but the state (the resource is gone) is the same.
Second, most of the interesting actions in automation are POSTs. Creating records, sending messages, charging cards and starting jobs are typically not idempotent by default. That is why they need deliberate protection.
Beyond HTTP, the idea generalises. An operation is idempotent if repeating it does not compound its effect. "Set the status to shipped" is idempotent. "Add one to the counter" is not. "Insert this row if no row with this key exists" is idempotent. "Insert this row" is not.
Where duplicates come from
You cannot design out duplicates if you do not know where they originate. In automation there are at least eight common sources.
1. Webhook redelivery. Senders retry when they do not get a fast success response. Stripe's documentation states plainly that webhook endpoints might occasionally receive the same event more than once, that it retries failed deliveries for up to three days with exponential backoff in live mode, and that it does not guarantee event ordering. It recommends tracking event IDs to identify duplicates.
2. Your own retries. Workflow tools and code libraries retry on timeouts and 5xx errors. If the first attempt succeeded but the response was lost, the retry repeats the action.
3. Timeouts that hide success. A client gives up after 30 seconds. The server finishes at second 31. From the client's view, the request failed; in reality, it worked.
4. Users double-submitting. Someone clicks Submit twice, or refreshes after a form post.
5. Overlapping triggers. A schedule fires while the previous run is still executing, or two triggers (a webhook and a polling check) both react to the same event.
6. Manual re-runs. Someone re-executes a failed workflow from the beginning, repeating steps that had already succeeded.
7. Upstream data quirks. The source system emits two events for one change, or two different IDs for what is logically the same record. Stripe notes that in some cases two separate event objects are generated and suggests using the ID of the object inside the event along with the event type to detect them.
8. Restarts and deploys. A worker restarts mid-run and the queue redelivers the message that was in flight. Many queues offer at-least-once delivery, which by design can repeat messages.
The common thread: distributed systems cannot cheaply guarantee that an action happens exactly once. What you can do is make repeating an action harmless, which gets you the same practical result, often described as "effectively once."
Why AI workflows make this worse
Automations with model calls have three properties that make duplicate handling more important.
Model steps are slow and flaky. They take seconds rather than milliseconds, they get rate-limited, and they sometimes time out. More timeouts mean more retries.
Model steps are non-deterministic and costly. Re-running a model call may give a different answer, and it costs money each time. If a retry re-generates a reply and sends it, the customer might receive two different emails. Even when duplicates are harmless in effect, they waste tokens.
Model steps sit before consequential actions. The typical chain is: receive event, call model, act on the result. Duplicates in the first step propagate to actions.
The design principle that follows: run the expensive, non-deterministic step once per logical item, store its result, and make every later step idempotent given that stored result.
The core pattern: an idempotency key
The practical tool for making non-idempotent operations safe is the idempotency key: a unique identifier for the logical operation, checked before the operation runs.
Stripe's API documentation describes the mechanism clearly. For POST requests you supply an Idempotency-Key header. Stripe saves the resulting status code and body of the first request made with that key, whether it succeeded or failed, and returns the same result for later requests with the same key, including 500 errors. Keys can be up to 255 characters; Stripe suggests V4 UUIDs or another random string with enough entropy; it advises against using sensitive data such as email addresses as keys; keys can be pruned after at least 24 hours; and the system compares incoming parameters against the original request and errors if they differ, to prevent accidental misuse. GET and DELETE requests do not need keys because they are idempotent by definition.
You can apply the same idea inside your own workflows, whether or not the remote API supports keys.
Choosing the key
The right key identifies the logical operation, not the attempt. Good choices:
A stable ID from the source. The webhook's event ID, the email's Message-ID header, the form submission ID, the row's primary key.
A deterministic composite. customer_id + invoice_number, or order_id + "welcome_email". The same logical action always produces the same key.
A hash of the meaningful content. Useful when no ID exists: hash the normalised fields that define uniqueness.
Bad choices:
A timestamp or random value generated at run time. Every retry gets a new key, which defeats the purpose.
Values that change between attempts. Anything that includes "now" or an attempt counter.
Sensitive personal data. Keys end up in logs and tables.
A helpful test: if the same event were delivered twice, would your key be identical both times? If yes, it is a good key.
The check-then-act trap
A naive implementation looks like this:
Look up whether the key has been processed.
If not, perform the action.
Record the key.
This has a race. Two concurrent deliveries can both find the key missing at step 1, and both proceed. The check and the record must be atomic. The most reliable way is to let a database enforce uniqueness.
Pattern 1: a processed-events table with a unique constraint
Create a table whose primary key is the idempotency key, and insert into it as the first step, relying on the constraint to reject duplicates.
create table processed_events (
event_key text primary key,
received_at timestamptz not null default now(),
status text not null default 'processing',
result jsonb
);
The claim step is a single atomic statement:
insert into processed_events (event_key)
values ($1)
on conflict (event_key) do nothing
returning event_key;
If a row comes back, this delivery won the claim and should do the work. If nothing comes back, another delivery already claimed it, so skip, or return the stored result.
Three details make this robust.
Record status, not just presence. If the worker crashes after claiming but before finishing, the key exists but the work is incomplete. Track processing, done and failed, and a claimed_at timestamp. A separate sweep can re-open claims stuck in processing for longer than a reasonable timeout, so the work is retried instead of lost.
Store the result. Saving the output of the expensive step (the model's extracted JSON, the created record's ID) means a replay can return the same result without calling the model again. That is both cheaper and consistent.
Set a retention policy. Keep keys long enough to cover the longest plausible redelivery window. Stripe retries for up to three days in live mode and allows manual resends for weeks, so keep keys for at least that long for such sources; then prune.
In a visual tool such as n8n, you can implement the same thing with a database node (Postgres or Supabase) executing the insert-on-conflict statement, followed by a conditional node that stops the workflow when no row was returned.
Pattern 2: upserts instead of inserts
If the target is a database, prefer upsert semantics: insert if new, update if existing, keyed on a natural unique identifier.
insert into leads (source_id, email, name, category, updated_at)
values ($1, $2, $3, $4, now())
on conflict (source_id)
do update set
email = excluded.email,
name = excluded.name,
category = excluded.category,
updated_at = now();
Running this twice with the same source_id leaves one row. The unique constraint on source_id is doing the real work; the application code cannot forget it or race past it.
For CRM APIs, look for the equivalent: many offer create-or-update operations keyed on email or an external ID. Use those in preference to separate search-then-create calls, which reintroduce the race.
Pattern 3: make actions declarative, not incremental
Prefer operations that say what the final state should be over operations that describe a change.
Non-idempotent (incremental) | Idempotent (declarative) |
|---|
Add 1 to the follower count | Set the follower count to the value from the source |
Append this note to the record | Write the note under a key derived from the event ID |
Send a "welcome" email | Send the welcome email only if welcome_sent_at is null, then set it atomically |
Create an invoice | Create the invoice with external_id = order_id, rejecting duplicates |
Increase stock by 5 | Set stock to the count from the latest stocktake |
Where you must do something incremental, wrap it in the key check from pattern 1.
Pattern 4: separate the decision from the effect
A robust structure for AI workflows splits each item's processing into stages with stored outputs:
Receive and record. Save the raw event and claim its key. Return success to the sender quickly.
Enrich with the model. Call the model, validate the result, and store it against the item. If a stored result exists, skip the call.
Apply effects. For each external effect (create CRM record, send email, post to chat), check a per-effect flag or key, perform it, and record completion.
Each stage can be retried independently, because each is guarded by its own stored state. If the email step fails, you retry only the email, not the model call and not the CRM write. Stripe's advice to return a 2xx quickly before running complex logic fits stage one: acknowledge receipt, then process asynchronously, ideally through a queue, so a spike in deliveries does not overwhelm the handler.
This structure is closely related to a resumable state machine, where each item carries a status and the workflow moves it forward one stage at a time; resumable state machines for AI automation goes further into that design.
Pattern 5: outbox for side effects
Sending an email or calling an external API is a side effect you cannot roll back with a database transaction. A common approach is an outbox: instead of sending directly, write a row to an outbox table in the same transaction that updates your state, then have a separate worker send pending rows and mark them sent.
The guarantees are useful: if the transaction fails, no message is queued; if the worker crashes after sending but before marking, the message may be sent twice, so the send itself should also be idempotent, using an idempotency key if the provider supports it, or a provider-side deduplication ID.
You do not need to build this for every small workflow. It becomes worthwhile when the cost of a duplicate or a lost message is high, such as payments, legal notices and customer-facing notifications.
Idempotency for model calls
Language model APIs deserve special thought, because they are costly and non-deterministic.
Cache by input. For a given item and a given prompt version, store the model's output keyed by a hash of the input plus the prompt version. A retry or replay reads from the cache instead of calling the model.
Do not rely on identical output. Even with low randomness settings, outputs can vary, and providers may update models. If consistency matters, persist the first accepted output as the record of truth.
Retry only what failed. If a call times out, retry it, but do not re-run earlier stages. Use bounded retries with exponential backoff and jitter, and log every attempt. Gemini API error handling and retries covers the mechanics for one provider, and the principles apply broadly.
Beware of side effects inside tool calls. In an agent, a tool that sends an email or writes to a system is a side effect, and the model may call it twice, or a retry of the whole run may. Make each such tool idempotent by requiring a key argument derived from the task, or by checking state inside the tool. Because the model only requests tool calls and your code executes them, as OpenAI's function-calling documentation describes, your execution layer is the right place to enforce this.
Verifying webhook senders and duplicates together
Duplicate handling and authenticity checking belong together in a webhook receiver: first verify that the request really came from the expected sender, then check the idempotency key, then process. Doing it in the other order means an attacker can burn your keys or trigger actions with forged events. The security side, including signature verification and replay protection, is covered in webhook security for automation, and the mechanics of the n8n Webhook node are in n8n webhooks explained.
Note also that in n8n the Webhook node can respond immediately with a "workflow started" message or wait until the last node finishes. If the sender retries when it does not receive a timely response, responding immediately and processing afterwards reduces redelivery caused by slow model steps. Whichever mode you choose, keep your idempotency check at the very start of the processing path.
A worked example: lead intake, made duplicate-safe
Take the lead-intake workflow: a form submission hits a webhook, a model extracts fields, a CRM record is created, a reply is drafted, and a notification is sent.
Key. The form platform provides a submission ID, so the key is lead:{submission_id}.
Step 1: claim. Insert the key into processed_events with on conflict do nothing. If nothing is returned, respond 200 and stop.
Step 2: store the raw event. Save the original payload in a lead_events table, keyed by the same ID.
Step 3: extract. Check for a stored extraction for this submission. If present, use it. Otherwise, call the model, validate the JSON, and store it with the prompt version and a timestamp.
Step 4: CRM upsert. Use the CRM's create-or-update endpoint keyed on email, and store the returned CRM ID against the lead.
Step 5: draft reply. Check for an existing draft for this submission. If none, generate one and store it. It is a draft, so duplicates are not customer-visible in any case, but you still save the tokens.
Step 6: notify. Post to the chat channel only if notified_at is null, then set it. To make the check-and-set atomic, use a single update statement: update lead_events set notified_at = now() where id = $1 and notified_at is null returning id. Only the run that gets a row back posts the message.
Failure and replay. If the workflow dies after step 3, a re-run finds the claim, finds the stored extraction, skips the model, and continues at step 4. If someone re-executes it manually from the start, each step notices completed work and does nothing. If the form platform delivers the webhook three times, only the first proceeds.
The extra work is modest: a couple of tables and a few conditionals. The result is a workflow you can safely retry, replay and re-run.
Polling and scheduled workflows: cursors, not "everything new"
Not every automation is triggered by a webhook. Many poll a mailbox, an API or a table on a schedule, and they have their own duplicate hazards.
The "last hour" trap. A workflow that runs hourly and fetches "items from the last hour" will process items twice if a run is delayed and the windows overlap, and miss items if a run is skipped. Time windows are fragile.
Use a cursor instead. Store a watermark, such as the highest record ID or the latest updated_at value processed, and fetch items after it. Advance the watermark only after the batch is safely stored. If the run crashes halfway, the next run re-fetches from the old watermark, which is safe as long as processing each item is itself idempotent. Combining cursors with per-item keys gives you two layers of protection.
Overlap deliberately. A small overlap in the fetch window, for example fetching from a little before the watermark, protects against late-arriving records at the source. The per-item key then filters out the repeats.
Prevent concurrent runs. If a scheduled run can start while the previous one is still going, add a lock, such as a row in a locks table claimed with an insert that fails on conflict, or use the scheduler's setting that skips a run if one is active. Otherwise two runs will process the same items concurrently, which is exactly the race that keys are meant to resolve but which still wastes work.
Mailboxes need care. Marking a message as read is not a reliable "processed" flag, because a person might open it, or the mark might fail after the action succeeded. Prefer the message's unique identifier stored in your own table, and treat the mailbox state as a hint.
Partial failures and compensation
Real workflows have several effects, and any of them can fail after earlier ones succeeded. Suppose a workflow creates a CRM record, then creates a task, then posts a message. If the task creation fails permanently, you have a record without a task.
There are three sensible responses.
Retry until success, idempotently. For transient failures, retrying only the failed step is the default. The earlier steps are guarded, so they will not repeat.
Compensate. For permanent failures, undo earlier effects where that is possible and safe, such as deleting the record you just created, or marking it as incomplete. This is sometimes called a compensating action. It should itself be idempotent, since it may be retried.
Park for a human. When neither is appropriate, mark the item as failed with the reason and the state reached, and route it to a review queue. A person can then decide. This is often the right choice for rare, high-stakes failures, because automated cleanup written for a case that almost never happens is itself likely to be buggy.
Whichever you choose, the item's stored status should always reveal how far it got. If you can query "show me all leads that have a CRM record but no notification," you can find and repair partial failures instead of discovering them from customer complaints.
Idempotent email and message sending
Because duplicated customer messages are among the most visible failures, they deserve specific care.
Guard with a sent flag set atomically. Use the conditional update pattern, where only the process that successfully flips sent_at from null to a timestamp goes on to send.
Choose which failure you prefer. There is a genuine trade-off. If you claim the flag first and then send, a crash in between means the message is never sent, an at-most-once outcome. If you send first and record afterwards, a crash in between means the message is sent again on retry, an at-least-once outcome. For notifications where a duplicate is a nuisance and a miss is costly, at-least-once is usually right. For messages where a duplicate is harmful, such as payment demands, at-most-once with a monitored recovery process may be safer. Decide deliberately per message type, and write it down.
Use provider deduplication when available. Some email and messaging APIs accept a client-supplied identifier to suppress duplicates. If yours does, pass your key. If it does not, your own flag is the protection.
Keep drafts for the risky ones. While you are still building trust in a workflow, saving a draft for a person to send removes the duplicate-send risk for the customer entirely, at the cost of a human step.
Add a visible marker. Include a reference ID in the message body or a header. If a duplicate does slip through, support staff can spot it quickly and you can trace it in the logs.
A note on frameworks and tools
Different tools give you different building blocks, and it is worth checking what yours offers before writing your own.
Databases are the strongest option: unique constraints, insert ... on conflict, and conditional updates give you atomicity for free.
Workflow platforms usually offer some kind of data store, deduplication or "already seen" helper, and execution history. Read the documentation for your version, because behaviour and limits differ, and be aware that platform-level dedupe may be scoped to one workflow or one execution.
Queues typically offer at-least-once delivery, sometimes with deduplication windows or message IDs. Understand the semantics of the one you use.
Third-party APIs may support idempotency keys, as Stripe's does. Where they do, use them on every non-idempotent request; where they do not, protect the call yourself.
Whatever the tool, the principle is the same: identify the logical operation, claim it atomically, store what you produced, and make each effect safe to repeat.
Testing that your workflow is really idempotent
Idempotency you have not tested is a hope. These tests are cheap.
Duplicate delivery test. Send the same webhook payload two, three and ten times, sequentially and then concurrently. Assert that you end with one CRM record, one notification and one draft.
Concurrent delivery test. Fire duplicates at the same instant. The race conditions live here. If you see two records, the check-then-act trap has caught you.
Failure injection. Make each external call fail after it has actually succeeded (simulate a lost response), then let the retry happen. Confirm there is no duplicate effect. Also fail before the call and confirm the retry completes the work.
Crash test. Kill the worker mid-run, after the claim but before completion, and confirm a recovery process picks up the item exactly once.
Manual re-run test. Re-execute a completed run from the start and confirm nothing changes.
Out-of-order test. Deliver events in reverse order. Stripe says it does not guarantee delivery in the order events are generated, and does not recommend using timestamps to determine order or whether an event was already processed. Your handler should tolerate an "update" arriving before the "create," for example by fetching the current object state from the source API rather than trusting the event payload alone.
Long-delay test. Redeliver an event days after the original, as a provider's manual resend might, and confirm your retention window still recognises it.
Automate these as regression tests. Testing n8n workflows before production and testing LLM integrations with mocking cover practical approaches.
Observability for duplicates
Make duplicates visible so you can tune your design.
Log every claim attempt with the key and whether it won or was rejected.
Track the duplicate rate per source. A sudden rise suggests a sender problem or a timeout somewhere on your side.
Track items stuck in processing beyond a threshold.
Alert when the same key is rejected many times, which can indicate a retry storm.
Log which stage each retry resumed from.
Audit trails and decision logging and agent observability and tracing describe useful log designs.
Where you can skip it
Not every step needs a key. Reads are naturally idempotent: looking up a customer or fetching a document twice has no side effect. Every check also costs an extra database read, plus a write on success, which adds up at high volume. Spend that cost where a duplicate does real damage (a charge, a customer-facing message, a write another system depends on) and leave it off steps where repeating changes nothing, such as rewriting a cache entry with the value it already holds.
Common mistakes
Generating the key at run time. A new UUID per attempt provides no protection. Derive it from the source event.
Checking without atomicity. Look-up followed by insert races. Use a unique constraint or a single conditional statement.
Recording the key before the work but never recovering. A crash leaves the item permanently "claimed." Use statuses and a sweeper.
Recording the key after the work. A crash between the action and the record repeats the action. Claim first, complete after, and make the action itself idempotent where possible.
Deduplicating on the wrong field. Using email alone can merge two different people who share an address, and using a timestamp can miss true duplicates. Choose keys that match your business definition of "same."
Forgetting non-obvious effects. Sending a chat message, incrementing a counter and appending a row are all effects that need protection.
Retrying non-idempotent calls blindly. Only retry POSTs when you have a key or have verified that the effect did not occur.
Ignoring ordering. Do not assume events arrive in sequence.
Letting the model be the source of truth for identity. If the model decides whether two leads are the same person, treat that as a suggestion with review, not as the deduplication mechanism.
A short checklist
Every workflow has a stable key derived from the source event.
The claim is atomic, enforced by a unique constraint.
Claims have statuses and a recovery path for stuck items.
The expensive model result is stored and reused.
Every external effect is guarded by a key, a flag or a declarative operation.
Receipt is acknowledged fast, and heavy work happens asynchronously.
Ordering is not assumed.
Tests cover duplicate, concurrent, failure, crash and replay cases.
Duplicate rates and stuck items are monitored.
If you have a workflow that occasionally duplicates records or messages and want help making it safe to retry, get in touch.
Written by Abhishek Patoliya, Senior n8n Developer & Automation Architect, Surat, India. Verified n8n Creator.