Reliable ERP order delivery: retries, idempotency, and replay
Classify ERP delivery results, calculate bounded retry delays, and investigate lost responses before a repeated request creates another order.
In this guide
- Commerce accepted
- Durable command
- Provider attempt
- Reconcile outcome
- Confirmed delivery
The timeout after the order was created
The storefront accepts an order and an integration sends it to an ERP. The ERP commits successfully, but the response is lost. The sender sees a timeout. Retrying as a fresh order can create a duplicate; marking the order failed can abandon a successful transaction.
This is an unknown outcome, not proof of failure. Design for it before adding retry loops. Give each business command a stable identity and record delivery separately from storefront acceptance. The customer receipt should describe what commerce has accepted without pretending it knows every downstream result.
Establish the real trigger first
Identify which supported mechanism invokes the integration in the target Znode release. A legacy ERP touchpoint and a Data Exchanges custom processor have different contracts and lifecycles. A project method called 'order created' is not evidence that either mechanism invokes it automatically.
Trace one non-production order from the user action into the actual adapter. Capture the delivery baseline before placing the order, then observe the new delivery record. Do not manually dispatch afterward and mistake that activity for automatic platform behavior. If testing a manual dispatch, label it explicitly and keep the two exercises separate.
At the integration boundary, persist the command before acknowledging it as queued. Scope its stable key to Store, operation, and order, and compare a payload fingerprint on repeated submissions. The durable store owns atomic uniqueness and lease transitions; RetryDecision supplies only the scheduling policy for those persisted attempts.
Classify delivery outcomes before scheduling
Call RetryDecision.Choose after the provider adapter has classified a completed attempt. Pass the number of attempts already made, the maximum allowed, and a jitterFraction between 0 and 1. Jitter varies the delay slightly so many failed deliveries do not all retry at the same instant.
The returned Decision has a named Action and optional Delay. Confirmed means complete. Permanent rejection or an exhausted temporary-retry budget needs attention. Unknown goes to reconciliation, where a worker or operator checks whether the provider applied the command. Only a confirmed temporary rejection receives RetryLater.
Delivery/RetryDecision.cs csharp
using System;
namespace Gcg.DeveloperExamples.Delivery
{
public enum Outcome
{
Confirmed, TemporaryRejection, PermanentRejection, Unknown
}
public enum NextAction
{
Complete, RetryLater, NeedsAttention, Reconcile
}
public sealed class Decision
{
public NextAction Action { get; }
public TimeSpan? Delay { get; }
public Decision(NextAction action, TimeSpan? delay)
{
Action = action;
Delay = delay;
}
}
public static class RetryDecision
{
public static Decision Choose(Outcome outcome, int attempts, int maximumAttempts,
double jitterFraction)
{
if (attempts < 1 || maximumAttempts < 1 ||
double.IsNaN(jitterFraction) || jitterFraction < 0 || jitterFraction > 1)
throw new ArgumentOutOfRangeException();
if (outcome == Outcome.Confirmed)
return new Decision(NextAction.Complete, null);
if (outcome == Outcome.Unknown)
return new Decision(NextAction.Reconcile, null);
if (outcome == Outcome.PermanentRejection || attempts >= maximumAttempts)
return new Decision(NextAction.NeedsAttention, null);
if (outcome != Outcome.TemporaryRejection)
throw new ArgumentOutOfRangeException(nameof(outcome));
double seconds = Math.Min(300, Math.Pow(2, Math.Min(attempts, 8)));
return new Decision(NextAction.RetryLater,
TimeSpan.FromSeconds(seconds * (0.75 + jitterFraction * 0.25)));
}
}
}
// Classify a timeout after submission as Unknown until provider reconciliation.
// Persist the selected action with the current delivery lease before scheduling.
Choose(Outcome.TemporaryRejection, 1, 3, 0.5) returns RetryLater with a 1.75-second delay. Choose(Outcome.Unknown, 1, 3, 0.5) returns Reconcile with no retry delay. Persist the action against the current delivery record before scheduling work. This policy decides what should happen next; the durable worker owns claiming, scheduling, and executing that work.
Retries need a provider agreement
An idempotency key identifies one intended business operation across repeated deliveries. Confirm whether the provider stores that key atomically with the result and rejects the same key when its payload changes. A local delivery record alone cannot prevent two external effects.
If the provider supports lookup by an immutable reference, use it to investigate a lost response. Otherwise define an operator reconciliation path. Keep invalid business data, temporary rejection, and uncertain submission as distinct outcomes so a retry loop cannot turn an unknown order into a duplicate.
Exercise commit-then-error and concurrent replay
Build a mock provider that saves an order and deliberately drops the response. Verify that the next attempt uses the same business identity and resolves the existing result according to the mock's declared contract. Then run the same failure scenarios against the selected provider's sandbox contract.
Run two workers against one due delivery, expire a lease, and attempt simultaneous operator replays. Ensure claims and transitions remain controlled. Replay should preserve the original command identity, require authorization, and record why it was requested. Test an altered payload under the same key and a nonretryable validation failure as well.
- Keep raw provider bodies out of public status responses.
- Record sanitized failure categories and correlation references.
- Distinguish a prepared integration from an exercised live transport.
Define completion in operational terms
Agree what 'delivered' means: a transport response, provider acceptance, or a confirmed external order identifier. Later shipment and invoice events are separate milestones. Expose those distinctions so operators know which system owns the next action.
Reliable delivery comes from recoverable decisions rather than optimistic labels. A durable command, a stable identity, a provider-specific reconciliation rule, and a controlled replay process make failures manageable. They also give sales and support teams a more accurate explanation than simply asking a buyer to place the order again.
References and further reading
Independent guidance from GCG. Znode is a trademark of its owner. Examples use fictional data and are not official platform documentation. Suggest a correction.