Recover from a timed-out write without creating duplicates
Read back a timed-out write using its original key and intended data, then reconcile a match or escalate uncertainty without issuing another write.
In this guide
- Intent with stable key
- Write attempt
- Timeout
- Authorized readback
- Reconciled result
- Wait or manual review
Unknown is not failed
A client can time out after the server committed a record but before the response reached the caller. Retrying the same write without a stable key can create duplicates. Treat the state as unknown until the system can establish whether the requested business change exists. This is different from a confirmed validation error or a confirmed authorization denial.
Choose a stable business key before the first write. It may be a client-generated request ID accepted by the target, an external reference with a uniqueness constraint, or a project-owned delivery record that maps one intended action to one target record. A random retry identifier created after timeout cannot reconcile the original attempt.
- Confirmed success
- Confirmed rejected
- Unknown outcome
- Confirmed conflict
- Reconciled existing result
Make reconciliation a first-class flow
Keep WriteIntent in the durable operation record before the first write. StableKey identifies the intended action, PayloadHash identifies its exact meaningful input, and ReviewAfter is the time to escalate an unresolved result. RecoveryScope comes from server authorization.
RecoverAsync asks IWriteReadback for up to two committed results with that key. Each StoredWrite must match tenant, account, key, and payload hash. The returned RecoveryResult is Reconciled, AwaitReconciliation, or ManualReview. This service performs reads only and never issues another write.
Run recovery from a bounded worker with backoff and per-request timeouts. The gateway must report completed business effects rather than merely queued requests. Build the lowercase SHA-256 fingerprint from the same versioned field representation before writing and when interpreting committed readback.
WriteRecoveryService.cs csharp
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
namespace Gcg.Examples.Recovery;
public enum RecoveryResult
{
Reconciled, AwaitReconciliation, ManualReview
}
public sealed record RecoveryScope(Guid TenantId, Guid AccountId, Guid ActorId);
public sealed record WriteIntent(Guid StableKey, string PayloadHash, DateTimeOffset ReviewAfter);
public sealed record StoredWrite(Guid TenantId, Guid AccountId, Guid StableKey, string PayloadHash);
public sealed record ReadbackPage(IReadOnlyList<StoredWrite> Rows, bool HasMore);
public interface IRecoveryAuthorization
{
Task DemandReadAsync(RecoveryScope scope, Guid stableKey, CancellationToken ct);
}
public interface IWriteReadback
{
// Project adapter: fixed destination, authorized tenant/account, exact stable key.
// The result describes committed business effects, not just a queued command.
Task<ReadbackPage> FindAsync(RecoveryScope scope, Guid stableKey, int take, CancellationToken ct);
}
public sealed class WriteRecoveryService
{
private readonly IWriteReadback readback;
private readonly IRecoveryAuthorization authorization;
private readonly TimeProvider clock;
public WriteRecoveryService(
IWriteReadback readback, IRecoveryAuthorization authorization, TimeProvider clock)
{
this.readback = readback ?? throw new ArgumentNullException(nameof(readback));
this.authorization = authorization ?? throw new ArgumentNullException(nameof(authorization));
this.clock = clock ?? throw new ArgumentNullException(nameof(clock));
}
// Both scope and intent come from authenticated server state and the durable ledger.
// ReviewAfter is an escalation deadline, never proof that the original write stopped.
public async Task<RecoveryResult> RecoverAsync(
RecoveryScope scope, WriteIntent intent, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(scope);
ArgumentNullException.ThrowIfNull(intent);
ct.ThrowIfCancellationRequested();
if (scope.TenantId == Guid.Empty || scope.AccountId == Guid.Empty ||
scope.ActorId == Guid.Empty || intent.StableKey == Guid.Empty ||
!IsSha256(intent.PayloadHash))
throw new ArgumentException("A valid scope and durable write intent are required.");
await authorization.DemandReadAsync(scope, intent.StableKey, ct);
ReadbackPage page;
try
{
page = await readback.FindAsync(scope, intent.StableKey, 2, ct)
?? throw new InvalidDataException("Missing readback response.");
}
catch (TimeoutException) { return Pending(intent); }
catch (OperationCanceledException) when (!ct.IsCancellationRequested) { return Pending(intent); }
catch (HttpRequestException ex) when (ex.StatusCode is null) { return Pending(intent); }
if (page.Rows is null || page.Rows.Count > 2 || (page.Rows.Count == 0 && page.HasMore))
throw new InvalidDataException("Invalid bounded readback page.");
if (page.Rows.Count == 0)
return Pending(intent);
if (page.Rows.Count != 1 || page.HasMore)
return RecoveryResult.ManualReview;
var row = page.Rows[0];
if (row is null || row.TenantId != scope.TenantId || row.AccountId != scope.AccountId ||
row.StableKey != intent.StableKey ||
!string.Equals(row.PayloadHash, intent.PayloadHash, StringComparison.Ordinal))
return RecoveryResult.ManualReview;
return RecoveryResult.Reconciled;
}
private RecoveryResult Pending(WriteIntent intent) => clock.GetUtcNow() >= intent.ReviewAfter
? RecoveryResult.ManualReview : RecoveryResult.AwaitReconciliation;
// Hashes cover a versioned canonical payload with ownership fields; lowercase hex.
private static bool IsSha256(string value)
{
if (value is null || value.Length != 64)
return false;
foreach (char ch in value)
if (!((ch >= '0' && ch <= '9') || (ch >= 'a' && ch <= 'f')))
return false;
return true;
}
}
After a lost response, one matching committed row returns Reconciled. No visible row before ReviewAfter returns AwaitReconciliation; the original write may still be running. No row at or after the deadline returns ManualReview, not permission to resend. Duplicate rows or different intended data also require ManualReview. The supplemental fingerprint helper shows how to make field ordering and ownership part of the comparison.
Use a simulator before a real endpoint
A simulator should deliberately commit a record and then drop the response. It should also simulate a write that fails before commit, delayed visibility, duplicate submission, transient unavailability, and a conflicting existing record. These cases make it possible to test user messaging and recovery code without inferring behavior from production incidents.
Record a sanitized correlation ID, stable key hash or safe reference, state transition, and elapsed time. Do not log raw payloads, access tokens, or customer identifiers. A support view should make unknown and manual-review states visible to authorized operators.
- Timeout after commit.
- Timeout before commit.
- Readback delayed by eventual consistency.
- Concurrent duplicate submission.
- Existing record with different intended data.
Set realistic limits
Put any automatic retransmission in a separate delivery policy backed by the provider's documented deduplication guarantee. Reuse the original key and payload within that guarantee's retention period. A lookup that currently finds no row cannot prove the original request stopped.
Keep authorization and contract failures as errors, network uncertainty as pending recovery, and conflicts as operator work. Give operators a safe reference and a defined decision so a buyer is not encouraged to place another order while the first remains uncertain.
Verification checklist
Test every state transition and verify that a retry never bypasses account or store authorization. Confirm that the user receives a clear outcome and that the operator can trace a safe correlation ID. Revisit the design when the endpoint, storage, queue, or external partner contract changes.
Create a stable fingerprint for an application-owned note intent
Call Create before the first note-write attempt, using its tenant, account, stable key, and exact intended text. It writes those fields in a fixed order with schema 1 and operation set-support-note, then returns a lowercase SHA-256 fingerprint. Reconstruct the same fields from committed readback and call the same function for comparison. Changing an ownership value or meaningful text changes the fingerprint; hashing does not encrypt or authorize the data, so keep it in the protected recovery record.
Recovery/NoteIntentFingerprint.cs csharp
using System;
using System.IO;
using System.Security.Cryptography;
using System.Text.Json;
public static class NoteIntentFingerprint
{
public static string Create(Guid tenant, Guid account, Guid key, string text)
{
if (tenant == Guid.Empty || account == Guid.Empty || key == Guid.Empty ||
text is null || text.Length > 2000)
throw new ArgumentException("A scoped note intent is required.");
using var bytes = new MemoryStream();
using (var json = new Utf8JsonWriter(bytes))
{
json.WriteStartObject();
json.WriteNumber("schema", 1);
json.WriteString("operation", "set-support-note");
json.WriteString("tenant", tenant.ToString("D"));
json.WriteString("account", account.ToString("D"));
json.WriteString("key", key.ToString("D"));
json.WriteString("text", text);
json.WriteEndObject();
}
return Convert.ToHexString(SHA256.HashData(bytes.ToArray())).ToLowerInvariant();
}
}
References and further reading
Bring your next engineering question.
Need a resilient integration recovery design?
Independent guidance from GCG. Znode is a trademark of its owner. Examples use fictional data and are not official platform documentation. Suggest a correction.