Debugging on Brian Faust

Debugging Ends with an Explanation, Not a Green Build

Thu, 01 Dec 2016 00:00:00 +0000

A cryptography-heavy application held its CPUs near 100 percent. We found several cryptographic paths worth improving, which made the diagnosis feel obvious: expensive mathematics was consuming the machine.

Then we measured the log stream. The application was writing roughly a thousand lines per second to local disk.

Reducing emitted logging while leaving the application work unchanged returned CPU use to roughly 10–20 percent. The cryptographic findings were real. They were not the dominant cause of the observed saturation.

Test Cannot Prove Production When It Runs Different Rules

Sat, 01 Oct 2016 00:00:00 +0000

A label-layout change behaved correctly in test and disrupted document distribution in production. Both environments ran the same revision. The input looked equivalent. The test result was still incapable of predicting the production result.

The difference was deliberate. Test loaded one default and production loaded another. Users already had a way to select the layout they needed, but some had come to rely on the accidental production default. A new recommendation passed against the test rule while production continued through a different branch.

A Correlation Identifier Is a Boundary Contract

Thu, 01 Sep 2016 00:00:00 +0000

A shipment request failed somewhere after payment and before its documents were ready. The application, queue worker, and remote adapter all had logs. None of them shared an identifier.

The search began with timestamps, account details, and values which looked related. Payment was a plausible lead. So were several nearby remote calls. Some trails ended only after hours of reading because they belonged to different requests which happened at nearly the same time.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

The payment theory was attractive because of history, not because it explained the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Production Bugs Can Live in the Data, Not the Code

Mon, 01 Feb 2016 00:00:00 +0000

A delivery integration returned a label, but the application could not save it. The same persistence path had passed with the fixtures used elsewhere.

The failed label was not malformed. It was simply larger than the database column could hold after Base64 encoding.

That distinction changes the investigation. Comparing deployed revisions would find nothing because the revision was not the variable. Repeating a familiar fixture would also find nothing because the fixture did not cross the same storage boundary. The investigation had treated “same code” as “same experiment” while leaving the input that decided the result uncontrolled.

Reproduce the Failure Before Repairing It

Fri, 01 Jan 2016 00:00:00 +0000

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not yet a useful reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.