How to Prevent Duplicate Workflow Executions with Idempotency

Key takeaways

1. Retries are normal: webhooks, queues, schedulers, and users can deliver the same intent more than once.
2. Deduplicate business events, not HTTP requests: the key must represent the operation that should happen once.
3. Claim before side effects: a lookup followed later by an insert leaves a race window.
4. Store execution state: distinguish work that is in progress from work already completed.
5. Make the sink safe too: pass the same key to a provider that supports idempotency or enforce a unique constraint in your own database.

Why do duplicate workflow executions happen?

A sender can retry because it did not receive an acknowledgement. A queue can redeliver after a worker timeout. A person can double-submit a form. A scheduler can overlap with a slow run. None of those conditions proves that the first attempt failed before the external effect.

The dangerous window is simple: a workflow creates the invoice, sends the message, or writes the record, then crashes before it records success. A retry sees no completion marker and repeats the effect. Exactly-once delivery is rarely a safe assumption across independent systems; the practical objective is an idempotent business operation.

What should the idempotency key contain?

Prefer a stable event identifier supplied by the source, such as a webhook event ID, booking ID, order ID, or payment intent ID. If no trustworthy ID exists, derive a deterministic key from the fields that define one business operation—for example, tenant, operation type, source record, and version.

key = hash(tenantId + ":" + operation + ":" + sourceId + ":" + version)

Do not use a random UUID generated inside the receiving workflow; every retry would receive a new value. Do not hash volatile fields such as receive time. Names and email addresses alone can collide or change, so use source-system identifiers where possible. Store a payload hash beside the key and reject a reused key when consequential parameters differ.

Stripe’s official idempotent-request guidance documents the same safety property at an API boundary: reusing a key with different endpoint parameters produces an idempotency error. Provider semantics differ, so verify each API rather than assuming a shared standard.

What execution states should be recorded?

A conditional insert or unique key must claim the record atomically. A “search, then create” sequence is insufficient under concurrency because two workers can both observe absence. Store creation time, expiry, payload hash, state, attempt count, result reference, and the last verified error without storing secrets.

AWS Lambda Powertools describes an idempotency record with a key, payload hash, in-progress and completion state, expiry, and serialized response. That implementation is AWS-specific, but the state model is broadly useful.

How can this pattern be implemented in n8n?

1. Validate the trigger payload and extract a stable source event ID.
2. Construct the idempotency key before any email, payment, CRM write, upload, or publication.
3. Atomically claim the key in durable storage. Use a database uniqueness constraint when concurrency matters.
4. If the key is completed, return the stored result. If it is in progress, stop or defer.
5. Execute the external effect with the same key where the destination supports it.
6. Persist the provider’s external ID and mark completion.
7. On ambiguous failure, reconcile against the provider before retrying.

n8n documents that Data Tables can store markers to prevent duplicate runs, and its Data Table node can retrieve, insert, update, and upsert rows. The documentation also characterises Data Tables as light-to-moderate storage. For high-concurrency or financially consequential work, confirm atomic uniqueness and locking behavior in the chosen storage layer rather than treating an ordinary lookup as a lock.

Harrison’s documented Lead Generation Subscriber Agent performs duplicate lookup before record creation. That is useful data hygiene evidence; it is not presented here as proof of atomic idempotency under concurrent delivery.

How should duplicate protection be tested?

• Send the identical event twice sequentially and verify one external effect.
• Send the same event concurrently from two workers.
• Crash after the external call but before marking completion, then retry.
• Reuse a key with different parameters and verify rejection.
• Expire an in-progress record and test the reconciliation path.
• Retry provider timeouts, 429 responses, and 5xx responses without duplicating writes.
• Confirm stored responses are returned consistently and contain no secrets.

Use a sandbox or controlled target. Count actual effects at the destination—not merely successful workflow runs. A green execution log does not prove that only one invoice, message, row, or booking was created.