Idempotency Keys for Payment and Retry Safety

Retries are normal in production systems.

A mobile app may lose network after submitting a request. A browser may retry after a timeout. A backend job may retry after a transient failure. A payment provider may send the same webhook more than once.

The risk is that the system may process the same business action twice.

For normal reads, that is usually harmless. For payments, wallet credits, order creation, subscription changes, report generation, or any operation that changes money or entitlement, duplicate execution can become expensive and hard to explain.

That is where idempotency keys matter.

Problem

Imagine a payment or credit purchase flow.

A user clicks Pay. The frontend sends a request to the backend. The backend creates a payment intent, charges the user, updates a wallet, and returns a result.

But production is not a clean single path.

Several things can happen:

• the user double-clicks the button
• the browser retries after a timeout
• the mobile app loses connection after the backend succeeds
• the API gateway retries the request
• a background worker picks up the same job twice
• the payment provider sends duplicate webhooks
• the frontend does not receive the success response and asks again

From the user point of view, they performed one action.

From the system point of view, the same action may arrive multiple times.

If the backend is not designed for that, the system can create duplicate orders, charge twice, credit twice, generate two reports, or leave conflicting payment records.

Common mistake

The common mistake is trusting retries without tracking the original request.

The flow often starts like this:

POST /payments
  -> validate request
  -> call payment provider
  -> save payment
  -> credit wallet
  -> return success

Then retry logic is added around it:

If request fails, try again.

That sounds reasonable, but it is incomplete.

The retry does not know whether the first request failed before the payment, after the payment, after the wallet update, or only while sending the response back to the client.

Without a stable request identity, every retry looks like a new action.

Why it fails in production

Payment and entitlement flows fail in production because success and response delivery are not the same thing.

The backend may complete the operation, but the client may never receive the response.

For example:

Client sends payment request
  -> Backend creates payment
  -> Provider accepts payment
  -> Backend updates wallet
  -> Network fails before response reaches client

The client sees a timeout and retries.

If the backend creates a new payment on the second request, the user may be charged again. If the backend updates the wallet again from a duplicate webhook, the user may receive extra credit. If support later checks the logs, the story is hard to reconstruct because there are multiple records for what was logically one user action.

The same problem appears in other workflows:

• creating an order
• starting a paid report
• activating a subscription
• issuing a refund
• reserving inventory
• sending a payout
• applying a coupon

The technical problem is duplicate execution. The business problem is trust.

Better architecture

The better architecture is to give each write operation a stable idempotency key and store the request outcome.

The client or caller sends an idempotency key for the logical action:

POST /payments
Idempotency-Key: checkout_8f7c2b1d

The backend does not immediately execute the payment. First it checks whether this key has already been seen for the same user or account.

The backend can then decide:

• this is a new request, so process it
• this request is already in progress, so return a safe pending response
• this request already completed, so return the stored result
• this request failed structurally, so return the stored failure
• this key is being reused with different request data, so reject it

The key idea is simple:

Same logical request + same idempotency key = same result

The retry becomes safe because the backend can replay the outcome instead of repeating the side effect.

Suggested flow

A practical payment-safe flow looks like this:

Client creates an idempotency key
  -> Sends payment request with key
  -> API validates user, amount, and request body
  -> API stores request hash and status: Processing
  -> Payment workflow runs once
  -> Provider result is saved
  -> Wallet/order/subscription update is committed
  -> Final response is stored against the key
  -> Retry returns the stored response

For webhooks, the provider event id can play a similar role:

Webhook received
  -> Check provider event id
  -> If already handled, return success
  -> If new, process event
  -> Store event handling result

The important parts are the request key, the request fingerprint, the operation status, and the stored outcome.

C# example

This is a simplified C# example for an API endpoint. The exact persistence layer can be SQL Server, PostgreSQL, or another durable store. The pattern is what matters.

public enum IdempotencyStatus
{
    Processing,
    Completed,
    Failed
}

public sealed record IdempotencyRecord(
    string Key,
    string UserId,
    string RequestHash,
    IdempotencyStatus Status,
    int? HttpStatusCode,
    string? ResponseJson,
    DateTimeOffset CreatedUtc,
    DateTimeOffset UpdatedUtc
);

public sealed class PaymentService
{
    private readonly IIdempotencyStore _idempotencyStore;
    private readonly IPaymentGateway _paymentGateway;
    private readonly IWalletStore _walletStore;

    public PaymentService(
        IIdempotencyStore idempotencyStore,
        IPaymentGateway paymentGateway,
        IWalletStore walletStore)
    {
        _idempotencyStore = idempotencyStore;
        _paymentGateway = paymentGateway;
        _walletStore = walletStore;
    }

    public async Task<ApiResult> CreatePaymentAsync(
        string userId,
        string idempotencyKey,
        PaymentRequest request,
        CancellationToken cancellationToken)
    {
        var requestHash = PaymentRequestHasher.Hash(request);

        var existing = await _idempotencyStore.GetAsync(
            userId,
            idempotencyKey,
            cancellationToken);

        if (existing is not null)
        {
            if (existing.RequestHash != requestHash)
            {
                return ApiResult.Conflict(
                    "This idempotency key was already used for a different request.");
            }

            if (existing.Status == IdempotencyStatus.Completed)
            {
                return ApiResult.FromStoredResponse(
                    existing.HttpStatusCode!.Value,
                    existing.ResponseJson!);
            }

            if (existing.Status == IdempotencyStatus.Processing)
            {
                return ApiResult.Accepted("The original request is still processing.");
            }
        }

        await _idempotencyStore.TryCreateProcessingRecordAsync(
            userId,
            idempotencyKey,
            requestHash,
            cancellationToken);

        try
        {
            var providerResult = await _paymentGateway.ChargeAsync(
                request.Amount,
                request.Currency,
                cancellationToken);

            await _walletStore.CreditAsync(
                userId,
                request.Amount,
                providerResult.ProviderPaymentId,
                cancellationToken);

            var response = new PaymentResponse(
                providerResult.ProviderPaymentId,
                request.Amount,
                "completed");

            await _idempotencyStore.MarkCompletedAsync(
                userId,
                idempotencyKey,
                statusCode: 200,
                responseJson: JsonSerializer.Serialize(response),
                cancellationToken);

            return ApiResult.Ok(response);
        }
        catch (Exception ex) when (PaymentFailureClassifier.IsTransient(ex))
        {
            await _idempotencyStore.MarkFailedAsync(
                userId,
                idempotencyKey,
                "Transient payment failure",
                cancellationToken);

            throw;
        }
    }
}

In a real system, the store methods should use database constraints or transactions so two concurrent requests with the same key cannot both start processing.

Storage strategy

The idempotency record should be stored durably, not only in memory.

A useful table shape looks like this:

| Field | Purpose | | --- | --- | | Key | The caller-provided idempotency key | | Scope | User, account, tenant, or merchant boundary | | Request hash | Detects accidental key reuse with different payloads | | Status | Processing, completed, failed, or expired | | Response status | HTTP status code returned for completed requests | | Response body | Safe response payload to replay | | Operation reference | Payment id, order id, wallet transaction id, or job id | | Created/updated timestamps | Debugging, cleanup, and expiry | | Expiry timestamp | Defines how long keys remain valid |

There should be a unique constraint on the scope and key:

unique(scope, idempotency_key)

For payment webhooks, store the provider event id separately:

unique(provider_name, provider_event_id)

This prevents the same provider event from applying the same business change twice.

Retry behavior

Retries should return the state of the original request.

If the first request completed, return the stored success:

Retry with same key
  -> return original 200 response

If the first request is still processing, return a clear pending response:

Retry while processing
  -> return 202 Accepted
  -> client polls status or waits

If the same key is reused with a different request body, reject it:

Same key + different amount
  -> 409 Conflict

If the provider call is uncertain, do not blindly create a new payment. First reconcile with the provider or continue the original operation using a stored provider reference where possible.

The retry policy should be boring and explicit:

• retry network and timeout failures
• replay completed responses
• reject mismatched payloads
• deduplicate provider webhooks
• avoid repeating non-idempotent side effects
• expire old keys only after the business risk window is acceptable

Tradeoffs

Idempotency adds storage and state management.

You need a durable record, unique constraints, request hashing, response replay, expiry rules, and careful handling of in-progress requests. It is more work than writing a controller action that immediately performs the payment.

But the tradeoff is worth it when the operation has business consequences.

Use idempotency for:

• payments
• refunds
• wallet credits
• subscription changes
• order creation
• paid report generation
• inventory reservations
• external API calls that should not repeat

For low-risk operations, the full pattern may be too much. For money, entitlement, and user trust, it is usually part of the baseline architecture.

Production checklist

Before shipping a payment or retry-sensitive API, I would check:

• Does every risky write operation accept an idempotency key?
• Is the key scoped to the user, account, tenant, or merchant?
• Is there a unique constraint on scope and key?
• Is the request body hashed and compared on retry?
• Are completed responses stored and replayed?
• Are in-progress retries handled clearly?
• Are provider webhook event ids deduplicated?
• Are wallet/order/payment updates transactional where possible?
• Is uncertain provider state reconciled before retrying?
• Are idempotency records expired only after a safe window?
• Are duplicate side effects observable in logs or audit trails?
• Does support have enough information to explain what happened?
• Are key mismatches rejected instead of silently processed?
• Are clients documented to reuse the same key for retries?

LinkedIn short version

Retries are not safe just because the client sends the same request again.

For payment and entitlement workflows, the first request may have succeeded even if the response never reached the client.

Without idempotency, a retry can create a second payment, second order, second wallet credit, or second report.

The safer pattern is:

Generate an idempotency key.
Store the request hash.
Run the operation once.
Save the final response.
Replay that response on retry.
Reject the same key with different payload.
Deduplicate provider webhooks.

Retry safety is not only about retrying failed calls. It is about making sure the same business action cannot accidentally happen twice.