Debugging Ends with an Explanation, Not a Green Build

Thu, 01 Dec 2016 00:00:00 +0000

A cryptography-heavy application held its CPUs near 100 percent. We found several cryptographic paths worth improving, which made the diagnosis feel obvious: expensive mathematics was consuming the machine.

Then we measured the log stream. The application was writing roughly a thousand lines per second to local disk.

Reducing emitted logging while leaving the application work unchanged returned CPU use to roughly 10–20 percent. The cryptographic findings were real. They were not the dominant cause of the observed saturation.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

The payment theory was attractive because of history, not because it explained the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Causality on Brian Faust

Debugging Ends with an Explanation, Not a Green Build

A Failure Timeline Can Eliminate the Best-Looking Suspect