Debugging Real Systems on Brian Faust

Debugging Ends with an Explanation, Not a Green Build

Thu, 01 Dec 2016 00:00:00 +0000

A cryptography-heavy application held its CPUs near 100 percent. We found several cryptographic paths worth improving, which made the diagnosis feel obvious: expensive mathematics was consuming the machine.

Then we measured the log stream. The application was writing roughly a thousand lines per second to local disk.

Reducing emitted logging while leaving the application work unchanged returned CPU use to roughly 10–20 percent. The cryptographic findings were real. They were not the dominant cause of the observed saturation.

A Release Gate Must Fail Before Incompatible Code Goes Live

Tue, 01 Nov 2016 00:00:00 +0000

A deployment completed, traffic reached the new code, and database queries started failing. The code expected columns or tables which did not exist.

The missing step was documented. After each manual deployment, somebody had to run a shell script which applied the pending migrations. The instructions were available, but the working knowledge remained concentrated in one operator. Other people followed the sequence for a while and eventually skipped it.

Running the migrations restored the application. The incident was not caused by an obscure database failure. A release had been allowed to report success before one of its prerequisites was true.

Test Cannot Prove Production When It Runs Different Rules

Sat, 01 Oct 2016 00:00:00 +0000

A label-layout change behaved correctly in test and disrupted document distribution in production. Both environments ran the same revision. The input looked equivalent. The test result was still incapable of predicting the production result.

The difference was deliberate. Test loaded one default and production loaded another. Users already had a way to select the layout they needed, but some had come to rely on the accidental production default. A new recommendation passed against the test rule while production continued through a different branch.

A Correlation Identifier Is a Boundary Contract

Thu, 01 Sep 2016 00:00:00 +0000

A shipment request failed somewhere after payment and before its documents were ready. The application, queue worker, and remote adapter all had logs. None of them shared an identifier.

The search began with timestamps, account details, and values which looked related. Payment was a plausible lead. So were several nearby remote calls. Some trails ended only after hours of reading because they belonged to different requests which happened at nearly the same time.

Retries Can Multiply the Original Failure

Mon, 01 Aug 2016 00:00:00 +0000

Every failed job went back to the queue for another attempt 60 seconds later. There was no terminal condition and no meaningful failure classification. A malformed request received the same treatment as a short provider outage.

During long outages, roughly 50,000 jobs could sit in the system and keep trying for days. The queue backend filled, workers spent their time reproducing known failures, and the original dependency problem became sustained load on our own application.

A Timeout Is Part of the Contract

Fri, 01 Jul 2016 00:00:00 +0000

One remote accounting service would disappear for long periods and sometimes return with an expired TLS certificate. Another integration scheduled maintenance during peak operating hours. Work which normally completed quickly could occupy application capacity while the dependency was unable to help.

A timeout would not have made either provider reliable. It would have answered a question our callers still needed answered:

How long may this attempt wait before its result is no longer useful?

That answer belongs to the operation’s contract. Leaving it to a client library’s default lets a transport decide how much of the customer’s time and our worker capacity one dependency may consume.

Silencing an Error Does Not Protect the Customer

Wed, 01 Jun 2016 00:00:00 +0000

A shipment-booking request returned without an error. The shipment appeared in the system, so the request looked successful. It had no label and no tracking number.

Persistence had failed after the remote booking completed. A broad catch swallowed the exception, emitted no report or log, and continued. The code met one operational request—do not show customers errors—by returning an object which the customer could not use.

The intent was reasonable. Customers prefer reliable bookings to error messages. But once the system can no longer produce the required result, silence does not create reliability. It converts a known failure into an ambiguous success and hands the investigation to the customer.

Runtime Configuration Is Process State

Sun, 01 May 2016 00:00:00 +0000

A queued job selected the settings for one business context by overwriting Laravel’s global configuration. The values were correct for that job. The problem was what happened afterward.

The worker stayed alive.

Only a small fraction of the queued job classes selected their context this way. A later job could read a key without first replacing it and inherit whatever the previous job had left behind. Nothing was wrong with the configuration file, environment variables, or deployment. The process contained valid values for the wrong work.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

The payment theory was attractive because of history, not because it explained the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Logging Is a Workload, Not a Side Effect

Tue, 01 Mar 2016 00:00:00 +0000

One system was writing roughly a thousand log lines per second to local disk. Debug records were part of that stream. The individual call sites were small; the combined stream was not.

It was not cheap in aggregate. Every record still had to be constructed, serialized, copied through handlers, written, rotated, and eventually removed or archived. Whether anyone queried it later did not change that work.

We made log levels configurable, kept debug output on internal nodes where it was useful, and added rotation and archival. The important shift was not from “many logs” to “few logs.” It was from treating logging as harmless commentary to treating it as a workload with a budget.

Production Bugs Can Live in the Data, Not the Code

Mon, 01 Feb 2016 00:00:00 +0000

A delivery integration returned a label, but the application could not save it. The same persistence path had passed with the fixtures used elsewhere.

The failed label was not malformed. It was simply larger than the database column could hold after Base64 encoding.

That distinction changes the investigation. Comparing deployed revisions would find nothing because the revision was not the variable. Repeating a familiar fixture would also find nothing because the fixture did not cross the same storage boundary. The investigation had treated “same code” as “same experiment” while leaving the input that decided the result uncontrolled.

Reproduce the Failure Before Repairing It

Fri, 01 Jan 2016 00:00:00 +0000

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not yet a useful reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.