A Failure Timeline Can Eliminate the Best-Looking Suspect

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

The payment theory was attractive because of history, not because it explained the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Read partial state as a sequence

A broken record is not only missing data. It is also evidence that some effects occurred and others did not.

For this flow, the intended order was:

create local shipment row
    -> authorize payment
    -> create remote booking
    -> receive tracking number and label
    -> persist local shipment details
    -> return a usable result

The observed result was:

payment completed
remote booking completed
local shipment row exists
tracking number missing
label missing
request continued

Those lines do not yet explain why persistence failed. They do locate the failure window. A theory about an earlier boundary must explain how later effects appeared despite that earlier failure.

That is the first useful mental model for an effect timeline:

last completed required effect
    < failure window <=
first missing required effect

Here, remote booking is the last completed required effect. Local persistence of the returned details is the first missing one. The failure window sits between them.

Mark what was observed and what was inferred

Timelines become fiction when every plausible step is written as fact. The historic logs and raw request are not retained, so the article cannot recover timestamps or exception text. The DBG-04 fixture instead keeps three kinds of statement separate:

The distinction matters because an inference can be reopened. If later evidence shows that a “completed” payment flag was written before authorization or that a remote booking reference could be copied from an earlier attempt, the ordering changes.

Calling every row an observation would hide those reversal conditions. Calling none of them evidence would waste what partial state can tell us.

Make each suspect predict the remaining state

A useful theory predicts more than the symptom which suggested it.

Suppose the flow is synchronous and each step requires the preceding result. Three theories then make different predictions:

The word “no” is conditional. It means incompatible with the declared flow and the supplied effects. It does not mean payment systems can never create misleading state.

The synthetic verifier encodes those predictions. It rejects a theory when that theory requires an effect to be absent but the effect is present, or requires an effect to be present but it is missing. This is not a simulation of the private application. It is a check that the written reasoning and the declared observations agree.

This is stronger than ranking suspects by reputation:

reputation: payment breaks often, so inspect payment first
prediction: if payment failed here, remote booking could not have completed
observation: remote booking completed
decision: reject the theory under the current flow model

History still matters when choosing initial hypotheses. It should not let a familiar failure ignore contradictory effects.

The model breaks when effects can arrive out of order

The simple table depends on a synchronous flow. Real systems often weaken that assumption.

A queue may persist the local row before payment settles. A retry may reuse a remote reference from an earlier attempt. A webhook may deliver a tracking number after the request has failed. A cache may show stale completion. A database transaction may roll back local state while the remote side effect remains.

Under those conditions, presence alone does not establish order. The timeline needs more dimensions:

effect identity: which attempt produced it?
recorded_at:     when did this system observe it?
effective_at:    when did the external effect occur?
durability:      can this record roll back or expire?
provenance:      which boundary wrote it?

This is where correlation and durable attempt records become necessary. They separate “a booking exists” from “this request created that booking.” A later article owns the mechanics of carrying that identity across boundaries. The timeline only states when missing identity prevents a strong conclusion.

The payment theory should be reopened if payment and booking can proceed independently, if the completion evidence belongs to another attempt, or if the apparent remote success is only a locally cached intention. The model is a way to expose those assumptions, not a license to skip them.

Later code preserved the same class of warning

A retained private change from a later flow provides a separate analogue. A local shipment row could be created before tracking and metadata persistence. An unexpected throwable in the later phase bypassed the cleanup path and left the bare row behind. The repair broadened cleanup to cover that failure class and reported when deletion itself failed.

Another retained change moved a tracking value earlier in a document-saving sequence. The previous ordering allowed some tracking state to exist while the main row and documents remained incomplete if the process died between writes.

Neither change proves the historic incident. Together they confirm the more general mechanism: a multi-effect operation can leave a state whose completed and missing pieces reveal the interval in which execution stopped.

They also show the limit of a timeline. Moving a write or broadening cleanup changes which partial states survive. It does not prove that all external and local effects are atomic. A remote booking can succeed even when every local write rolls back.

A timeline narrows the next measurement

A full trace would be better evidence if it already existed. Adding broad instrumentation everywhere can still be wasteful when the surviving effects have reduced the question to one boundary.

For the partial booking, the next useful evidence belonged around response persistence:

remote booking reference received
local transaction active
tracking persistence attempted
document persistence attempted
exception class and field
transaction outcome
cleanup outcome

Payment telemetry would remain useful for operating payment. It would not explain why a completed remote booking lacked local tracking and document state under the synchronous flow.

This article stops before prescribing the repair. Returning an error, rolling back local state, cancelling the remote booking, or preserving partial state are product and consistency decisions with different costs. The next article in that part of the series owns the customer-visible failure and cleanup contract.

The immediate decision is smaller: do not add evidence equally across every plausible suspect. First order the effects which survived. A theory that cannot produce that state under the declared flow has already lost its place at the front of the investigation.