Debugging Ends with an Explanation, Not a Green Build

A cryptography-heavy application held its CPUs near 100 percent. We found several cryptographic paths worth improving, which made the diagnosis feel obvious: expensive mathematics was consuming the machine.

Then we measured the log stream. The application was writing roughly a thousand lines per second to local disk.

Reducing emitted logging while leaving the application work unchanged returned CPU use to roughly 10–20 percent. The cryptographic findings were real. They were not the dominant cause of the observed saturation.

That result sounded backwards at first. The application existed to perform cryptographic work; logging looked like bookkeeping. The investigation ended only when the less impressive subsystem explained the symptom better than the more impressive one.

A plausible defect is not yet the cause

Complex code offers an endless supply of suspicious details:

an expensive algorithm
an unnecessary allocation
a slow query
a lock with broad scope
an unbounded log path

Finding one proves that the detail exists. It does not prove how much of the reported symptom it caused.

The cryptographic work had a credible mechanism: more computation should consume more CPU. It also contained valid optimization opportunities. Those facts justified improvement, but they did not rank it against logging.

The logging hypothesis had its own mechanism:

construct records
format and serialize values
coordinate writers
make filesystem calls
copy bytes
rotate and retain files
repeat roughly 1,000 times per second

The mechanism made logging plausible. The intervention made it dominant.

This distinction matters because code review is biased toward what looks difficult. A cryptographic routine attracts attention. A logger call looks small. Production cost is accumulated execution, not visual complexity.

Write predictions before changing the system

Competing hypotheses become useful when each predicts a different observation.

For the saturated system:

The table does not identify the cause. It prevents every result from being interpreted as confirmation.

Without predictions, an investigator can optimize cryptography, observe some improvement, and declare victory even if logging still consumes most of the machine. Any busy system will reward enough random performance work with a positive number.

A useful experiment asks which hypothesis could have produced this particular change.

Change one causal variable when the system permits it

The decisive observation in the supplied incident was not merely that CPU fell. It was that emitted logging changed while the cryptographic application work did not.

The counterfactual is simple:

if cryptographic work were the dominant cause
and cryptographic work remained unchanged
then CPU should remain near the saturated state

It did not. CPU returned to the approximate 10–20 percent range.

That does not prove logging was the only cost in the process. It supports a narrower claim: reducing the logging workload removed most of the observed CPU saturation under the conditions remembered from that incident.

Single-variable interventions are often difficult in production. A deployment may include several fixes. Traffic may move. A host may restart. Caches may be warm on one run and cold on another. When changes cannot be isolated, narrow the conclusion:

strong: changing only log emission removed the saturation
weaker: a release containing three changes removed the saturation
invalid: the most interesting change in that release caused the improvement

The strength of the explanation should follow the isolation of the evidence.

Before-and-after numbers need provenance

The figures in this incident are recollections:

• CPU was sitting near 100 percent;
• the stream was roughly 1,000 local-disk lines per second; and
• CPU settled around 10–20 percent after logging was reduced.

There is no retained telemetry series, machine profile, request-rate graph, benchmark, deployment record, or exact log-volume sample available for this article. The values should not acquire extra precision through repetition.

They support the shape and scale of the observation. They do not support:

• an exact percentage improvement;
• an exact attribution between CPU, disk, locks, serialization, and runtime;
• a universal cost for one thousand log lines per second;
• a claim that traffic and host conditions were perfectly identical; or
• a regression threshold for another system.

This honesty does not erase the incident. It bounds it.

If the evidence still existed, I would want aligned time series for request rate, CPU, disk throughput and wait, emitted records, bytes written, and the deployment or configuration event. A profile would help separate formatting, locking, runtime, and filesystem work. Those measurements could sharpen the mechanism. We do not have them, so the article does not pretend that we do.

A secondary finding can remain worth fixing

Rejecting cryptography as the dominant cause does not make its inefficiencies imaginary.

An investigation can produce several valid outcomes:

dominant cause: explains most of the reported symptom
contributing cause: measurably worsens the symptom
latent defect: wrong but not active in this incident
optimization opportunity: useful without being a defect
false lead: evidence does not support the proposed mechanism

This vocabulary avoids a destructive binary. Teams often defend the first hypothesis because abandoning it seems to discard useful work. It is better to keep the finding and change its rank.

The cryptographic opportunities could enter the backlog with their own expected benefit and benchmark. They should not inherit urgency or causal credit from the CPU incident after logging explained the saturation.

That separation also improves review. “This code can be faster” and “this code caused the outage” require different evidence.

A green check proves only its assertion

A build can pass after a performance repair while the causal question remains open. Unit tests may prove output equivalence. A benchmark may prove one function became faster. Neither necessarily proves that the production symptom changed.

Evidence should line up across three layers:

correctness:
    required behavior still works

mechanism:
    the changed path controls the suspected cost

symptom:
    the original operational observation changes as predicted

For this incident, a green functional suite would protect cryptographic behavior while log controls changed. An isolated logging benchmark could show that fewer emitted records do less local work. The decisive production evidence would still be CPU and log rate moving in the predicted directions under comparable demand.

No such retained regression exists. The later logging controls reduced the incident, but this article cannot claim that a test now prevents recurrence.

A test should be described by what it can catch:

level configuration test:
    catches debug emission when the declared level disables it

rate-budget test:
    catches a synthetic path exceeding a chosen event budget

production alert:
    detects sustained resource saturation

none of these alone:
    proves why a particular past incident happened

Correlation is not intervention

Production metrics often move together:

traffic rises
CPU rises
logging rises
disk writes rise
latency rises

Correlation narrows candidates. It does not say whether logging caused CPU, traffic caused both, or latency caused retries which increased all three.

An intervention adds stronger evidence because it asks the system to respond:

hold the business workload as stable as practical
change log emission
observe whether CPU follows

Even then, the conclusion inherits the experiment’s limits. Lower logging may reduce disk contention, allocation, formatter work, or locks. Without profiles, “logging workload” is a supported boundary; one precise internal mechanism is not.

This article also does not repeat the earlier logging-budget argument. That article owns event rate, payload size, destinations, levels, rotation, and retention as design inputs. Here, the same approximate event rate matters only because changing that workload changed the symptom and ranked the hypotheses.

Record the explanation before memory compresses it

A useful investigation record can be short:

symptom
scope and time window
competing hypotheses
prediction for each
intervention and changed variables
observed result
mechanism supported by the result
alternatives demoted or retained
uncertainties and missing evidence
regression or operational guard
reversal condition

The DBG-12 fixture validates that shape against the supplied incident. It accepts the logging explanation because the symptom, two hypotheses, single changed factor, post-intervention observation, mechanism, provenance, and limitations are present.

It rejects three weaker records:

green check only:
    no operational symptom after the change

correlation only:
    simultaneous CPU and log rate without an intervention

multi-variable release:
    logging, cryptography, and runtime all changed together

The fixture is a deterministic evidence validator, not a performance reproduction. It cannot prove the historic CPU values or logging rate. Its job is to prevent the explanation from silently dropping the parts which make the causal claim credible.

Stop when the explanation can lose

An explanation is useful only if another observation could overturn it.

The logging conclusion would weaken if CPU remained saturated after emission fell under comparable work, if profiles showed negligible logging work, or if the apparent improvement tracked a traffic collapse or host replacement instead. Cryptography would regain causal weight if changing it moved CPU while logging remained constant.

These reversal conditions matter even after recovery. They keep a successful intervention from becoming folklore.

The surprising part of the incident was not that logging consumes resources. It was that a visually ordinary subsystem explained the machine better than the sophisticated code beside it. The investigation became complete when the dominant cause, valid secondary finding, changed variable, observed response, and missing evidence could all be stated without contradiction.

A green build says the assertions passed. A debugging explanation says why the system failed, why this change affected it, what else remains true, and what future evidence would make us revise the story.