The dangerous workflow failure is ambiguous success

A clean failure is usually manageable. The API returns an error, the browser shows a rejected action, or a transaction never leaves the client. The workflow can stop and make the failure visible.

The harder case is an action that may have succeeded while its acknowledgement was lost.

A CRM accepts a lead and closes the connection before returning the record ID. A message provider sends an SMS but the response times out. A browser submits a form and navigation stalls. Retrying blindly may create two records, send two messages, or place two orders. Refusing to retry may silently abandon work that never completed.

This is not primarily a retry-policy problem. It is an identity and reconciliation problem.

Give the intended effect a stable identity

Before the consequential call, derive or assign an operation key that survives process crashes and redelivery. The same logical trigger must produce the same key. Persist it before attempting the side effect.

Where the destination supports idempotency keys, send that key. Where it does not, write a correlation value into a searchable field or maintain a local mapping to the destination's record identifier.

A random key generated on every attempt is not idempotency. It only labels retries as new operations.

Model unknown separately from failed

A timeout after submission is not evidence of failure. Record an explicit unknown state:

• prepared
• attempting
• unknown
• confirmed
• rejected
• needs_operator

The recovery path for unknown is reconciliation, not immediate replay.

Reconcile against the destination

Before another write, ask whether the intended effect already exists. Use the stable operation key, provider request ID, destination record ID, or a narrow combination of immutable fields. The lookup itself needs bounded retries and a terminal operator state.

If the destination cannot support a reliable lookup, the workflow may require a human decision rather than unattended retry. That limitation belongs in the design, not in an incident postmortem.

Test the acknowledgement-loss boundary

A useful acceptance test deliberately creates this condition:

1. allow the destination to accept the write;
2. drop or delay the success response;
3. restart or redeliver the job;
4. assert that reconciliation runs before any second write;
5. assert exactly one logical outcome at the destination;
6. retain evidence connecting both attempts to the same operation key.

Happy-path mocks rarely exercise this boundary. Fault injection must occur after acceptance but before acknowledgement reaches the worker.

Keep authority bounded during recovery

Recovery code should not gain broader authority than the original workflow. A retry must not change recipient, amount, account, environment, or payload because new context appeared in an error message or page. If the proposed effect changes, require a new operation identity and, for consequential actions, a new approval.

The design objective is not “retry until green.” It is: every logical trigger has at most one consequential outcome, every uncertain result is reconciled, and every unrecoverable ambiguity becomes an explicit operator decision.

ScopeLatch's free local worksheet can draft the initial replay, unknown-write, recovery, authority, and evidence tests without transmitting workflow inputs: npub1cm54hczx404knjszkyjcyg0jv9udvh2h7ekc9nwkuk5g5x258fcs4u4m45.nsite.lol/test-plan.html