Recover from a timed-out write without creating duplicates
Treat timeout as an unknown outcome. Reconcile an exact committed result, keep absent readback pending, and escalate ambiguity without issuing another write.
In this guide
Unknown is not failed
A client can time out after the server committed a record but before the response reached the caller. Retrying the same write without a stable key can create duplicates. Treat the state as unknown until the system can establish whether the requested business change exists. This is different from a confirmed validation error or a confirmed authorization denial.
Choose a stable business key before the first write. It may be a client-generated request ID accepted by the target, an external reference with a uniqueness constraint, or a project-owned delivery record that maps one intended action to one target record. A random retry identifier created after timeout cannot reconcile the original attempt.
- Confirmed success
- Confirmed rejected
- Unknown outcome
- Confirmed conflict
- Reconciled existing result
Make reconciliation a first-class flow
After an uncertain response, query the original stable key inside the same authorized tenant/account boundary. Reconcile only one committed business result whose key, ownership, and canonical payload hash match the stored intent. A queued command is evidence of acceptance, not proof that the business effect completed.
An empty readback does not prove that the write failed. The original request may still be running, and a committed result may become visible later. This helper returns AwaitReconciliation until the ledger's review deadline, then ManualReview. It never emits a retry instruction and never performs a write. A documented consistency window can guide polling, but elapsed time alone cannot establish a safe new attempt.
Keep polling in a bounded worker with backoff and a per-call timeout. ReviewAfter is a server-owned escalation deadline. The lowercase SHA-256 value covers a versioned canonical representation of the intended fields, including ownership. Store that fingerprint before the first attempt and compare it using the same canonicalization on readback.
WriteRecoveryService.cs csharp
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
namespace Gcg.Examples.Recovery;
public enum RecoveryResult { Reconciled, AwaitReconciliation, ManualReview }
public sealed record RecoveryScope(Guid TenantId, Guid AccountId, Guid ActorId);
public sealed record WriteIntent(Guid StableKey, string PayloadHash, DateTimeOffset ReviewAfter);
public sealed record StoredWrite(Guid TenantId, Guid AccountId, Guid StableKey, string PayloadHash);
public sealed record ReadbackPage(IReadOnlyList<StoredWrite> Rows, bool HasMore);
public interface IRecoveryAuthorization
{
Task DemandReadAsync(RecoveryScope scope, Guid stableKey, CancellationToken ct);
}
public interface IWriteReadback
{
// Project adapter: fixed destination, authorized tenant/account, exact stable key.
// The result describes committed business effects, not just a queued command.
Task<ReadbackPage> FindAsync(RecoveryScope scope, Guid stableKey, int take, CancellationToken ct);
}
public sealed class WriteRecoveryService
{
private readonly IWriteReadback readback;
private readonly IRecoveryAuthorization authorization;
private readonly TimeProvider clock;
public WriteRecoveryService(
IWriteReadback readback, IRecoveryAuthorization authorization, TimeProvider clock)
{
this.readback = readback ?? throw new ArgumentNullException(nameof(readback));
this.authorization = authorization ?? throw new ArgumentNullException(nameof(authorization));
this.clock = clock ?? throw new ArgumentNullException(nameof(clock));
}
// Both scope and intent come from authenticated server state and the durable ledger.
// ReviewAfter is an escalation deadline, never proof that the original write stopped.
public async Task<RecoveryResult> RecoverAsync(
RecoveryScope scope, WriteIntent intent, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(scope);
ArgumentNullException.ThrowIfNull(intent);
ct.ThrowIfCancellationRequested();
if (scope.TenantId == Guid.Empty || scope.AccountId == Guid.Empty ||
scope.ActorId == Guid.Empty || intent.StableKey == Guid.Empty ||
!IsSha256(intent.PayloadHash))
throw new ArgumentException("A valid scope and durable write intent are required.");
await authorization.DemandReadAsync(scope, intent.StableKey, ct);
ReadbackPage page;
try
{
page = await readback.FindAsync(scope, intent.StableKey, 2, ct)
?? throw new InvalidDataException("Missing readback response.");
}
catch (TimeoutException) { return Pending(intent); }
catch (OperationCanceledException) when (!ct.IsCancellationRequested) { return Pending(intent); }
catch (HttpRequestException ex) when (ex.StatusCode is null) { return Pending(intent); }
if (page.Rows is null || page.Rows.Count > 2 || (page.Rows.Count == 0 && page.HasMore))
throw new InvalidDataException("Invalid bounded readback page.");
if (page.Rows.Count == 0) return Pending(intent);
if (page.Rows.Count != 1 || page.HasMore) return RecoveryResult.ManualReview;
var row = page.Rows[0];
if (row is null || row.TenantId != scope.TenantId || row.AccountId != scope.AccountId ||
row.StableKey != intent.StableKey ||
!string.Equals(row.PayloadHash, intent.PayloadHash, StringComparison.Ordinal))
return RecoveryResult.ManualReview;
return RecoveryResult.Reconciled;
}
private RecoveryResult Pending(WriteIntent intent) => clock.GetUtcNow() >= intent.ReviewAfter
? RecoveryResult.ManualReview : RecoveryResult.AwaitReconciliation;
// Hashes cover a versioned canonical payload with ownership fields; lowercase hex.
private static bool IsSha256(string value)
{
if (value is null || value.Length != 64) return false;
foreach (char ch in value)
if (!((ch >= '0' && ch <= '9') || (ch >= 'a' && ch <= 'f'))) return false;
return true;
}
}
Use a simulator before a real endpoint
A simulator should deliberately commit a record and then drop the response. It should also simulate a write that fails before commit, delayed visibility, duplicate submission, transient unavailability, and a conflicting existing record. These cases make it possible to test user messaging and recovery code without inferring behavior from production incidents.
Record a sanitized correlation ID, stable key hash or safe reference, state transition, and elapsed time. Do not log raw payloads, access tokens, or customer identifiers. A support view should make unknown and manual-review states visible to authorized operators.
- Timeout after commit.
- Timeout before commit.
- Readback delayed by eventual consistency.
- Concurrent duplicate submission.
- Existing record with different intended data.
Set realistic limits
Permit automatic retransmission only through a separate delivery policy backed by the target's documented idempotency guarantee, using the original key and payload within its retention period. Another safe route requires authoritative proof that the original attempt is terminal with no effect, together with protection against concurrent submissions. A business-key lookup by itself does not supply either guarantee.
Resolve unknown outcomes through authorized reconciliation. A transport timeout remains pending, an authentication or contract failure remains an error, and conflicting records require review. Include these states in the operator workflow so a buyer is never encouraged to create another order while the first remains uncertain.
Verification checklist
Test every state transition and verify that a retry never bypasses account or store authorization. Confirm that the user receives a clear outcome and that the operator can trace a safe correlation ID. Revisit the design when the endpoint, storage, queue, or external partner contract changes.
References and further reading
Bring your next engineering question.
Need a resilient integration recovery design?
Independent guidance from GCG. Znode is a trademark of its owner. Examples use fictional data and are not official platform documentation. Suggest a correction.