Index

Debugging Real Systems

Reproduce the Failure Before Repairing It

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not yet a useful reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.

For this failure, the smallest useful contract is almost embarrassingly plain: two accepted request executions must receive two non-empty, distinct tracing identifiers.

The identifier answered the wrong question

The old strategy derived a tracing identifier from the parsed JSON payload. That sounds reasonable if the identifier means “same operation.” It is wrong when the identifier means “this execution.”

Browser forms made the mismatch visible. Two form submissions could contain different fields while the JSON parser exposed an empty object for both:

ExecutionForm dataParsed JSONDerived identifier
firstone error and page{}hash of {}
secondanother error and page{}hash of {}

A cryptographic hash does not rescue the model. Identical input should produce an identical digest. The hash did its job; the application had asked it the wrong question.

The DBG-01 reproduction keeps that distinction executable:

const contentIdentifier = (request) =>
  sha256(JSON.stringify(request.parsedJson))

const first = contentIdentifier({ parsedJson: {} })
const second = contentIdentifier({ parsedJson: {} })

assert.equal(first, second)

That equality is not presented as a surprising property of SHA-256. It is the failure mechanism. Request details disappeared before the identifier was created, so two executions collapsed onto the same input.

Keep the first red test small

The retained regression exercised the real HTTP path twice and inspected the identifiers attached to the resulting reports. That boundary matters because the original problem crossed request parsing, middleware, application context, and report creation.

The assertion itself remained narrow:

first request  -> accepted -> identifier A
second request -> accepted -> identifier B

A is not empty
B is not empty
A is not B

It did not need a log backend, a dashboard, a queue, or a remote collector. Those components matter when proving that an identifier survives the whole request path. They do not help distinguish a content-derived collision from an execution-owned identifier.

This is where reproductions often become expensive for no gain. A production failure involves twelve services, so the test starts twelve services. The result may look realistic while adding clocks, networks, credentials, and unrelated state to the one observation that matters.

Start with the smallest boundary that can still produce the defect. Expand it when removing a component makes the failure disappear or changes the claim, not because the production diagram contains another box.

The repair changes identity, not hashing

The repair generated an application-owned UUIDv7 for each request rather than deriving request identity from its content. UUIDv7 was a practical choice, not the lesson. A correctly generated UUIDv4, ULID, or another sufficiently unique execution identifier could satisfy the same contract.

RFC 9562 defines UUIDv7 with a Unix timestamp field and random bits used to provide uniqueness. The W3C Trace Context Level 2 specification similarly treats a trace identifier as the identifier for a trace and prefers random generation over algorithms which derive global identity from request content.

The important change is semantic:

content-derived key: which operation does this payload describe?
execution identifier: which particular attempt are we observing?

Both can exist on the same request. In fact, they often should.

An idempotency key may deliberately remain stable when a client repeats one operation. The tracing identifier should still distinguish the first attempt from the retry, because their timing, failure point, deployment, and side effects may differ. Reusing one value for both jobs saves a field and destroys a distinction operators need.

The synthetic fixture therefore includes a control with one stable idempotency key and two execution identifiers. It proves that “same operation” and “same execution” are independent decisions. It does not claim that every payload needs idempotency or that a tracing identifier supplies deduplication.

A reproduction has a claim boundary

The local fixture proves four things:

  • different form requests can expose the same parsed JSON value;
  • hashing that value gives both executions the same identifier;
  • an execution-owned strategy can give them different identifiers; and
  • an idempotency key may remain stable while execution identifiers differ.

It does not prove that a UUID generator will never collide. It does not prove that identifiers propagate through queued work. It does not prove that every log line contains the active identifier, or that an observability backend indexes it correctly.

Those limits are useful. They stop a green check from becoming a claim about the entire tracing system.

The retained application regression provides stronger evidence: two requests travel through the HTTP path, create two reports, and expose two distinct identifiers. Even that test stops before the central logging system. Propagation is a separate contract and deserves separate evidence.

Escalate only when the smaller model stops explaining

A full traffic replay would be a credible alternative if the collision depended on a proxy rewriting headers, a framework parser selected by content type, or an interaction between middleware. It would also carry more privacy, state, and repeatability risk.

Here, the retained HTTP test and the smaller synthetic model agree on the first divergence: separate executions were assigned identity from the same empty JSON value. Adding the rest of production would not make that explanation more specific.

The decision reverses when the reduced case no longer produces the failure. If the local parser preserves distinct inputs but production still merges the traces, move outward one boundary at a time: ingress headers, middleware ordering, context storage, queue serialization, log enrichment, then backend indexing. Each expansion should answer a question created by the previous result.

Reproduction is not a demand to rebuild production on a laptop. It is a demand to make the failure disagree with the intended contract on command. Once that disagreement is small, repeatable, and bounded, the repair has something better than a plausible story to beat.