Quality Needs a Risk and Capacity Budget

An API change took about eight hours to release and another sixty hours to repair properly after it kept causing trouble.

The API was an active legacy system already associated with recurring defects, service interruptions, unreliable tests, difficult tooling, and delivery estimates which could miss by a factor of four or five. We still added work in a known performance-sensitive area because a customer or partnership needed it now.

The commercial logic was sound. The company existed to sell, acquire customers, and make money. The immediate value was legible. A failure six months away, conditional on traffic continuing to grow, was not.

The technical leadership problem was to make those two kinds of evidence comparable without pretending we could predict the future perfectly.

A forecast is not yet a shared fact

I was the CTO. I could see request growth, recurring defects, and shrinking capacity margin and conclude that the system was likely to fail if the trend continued. To a non-technical colleague, that conclusion could sound like an expensive opinion about software they had no practical way to inspect.

A colleague who spends the day selling can evaluate a customer commitment, contract value, and missed deadline directly. A dependency graph, query plan, or brittle test suite does not automatically translate into revenue at risk. A computer remains a black box until technical leadership supplies a useful window into it.

“This will collapse in six months” is therefore an incomplete communication, even when the forecast later proves correct. It needs premises a business decision can use:

• the traffic trend which reaches the current limit;
• the customer workflows affected at that limit;
• the expected frequency and duration of failure;
• the smallest intervention which changes the trajectory;
• the delivery displaced by that intervention; and
• the signals which would show that the forecast was wrong.

The CTO’s job is not merely to see around the technical corner. It is to show enough of the corner that other people can make a decision with it.

You cannot die on every hill

Pushing back has a cost. A technical leader who treats every quality concern as non-negotiable eventually makes the word “critical” meaningless. The company also cannot freeze delivery for six months every time a system contains debt. Some opportunities disappear while the architecture is being improved.

I could have pushed harder at points in this case. At other points, accepting the risk was reasonable. The decision depended on whether the likely failure was survivable, whether we could contain it, and what business we would give up to prevent it.

That means choosing among three responses rather than arguing only for or against a rewrite:

1. stop and repair when the likely consequence is existential or irreversible;
2. deliver with containment when the risk is material but manageable; or
3. accept the risk and observe it when prevention costs more than the likely failure.

Sometimes the incident becomes the evidence which a forecast could not supply. That does not mean engineering should manufacture a failure or quietly wait for vindication. It means recording the prediction, containing what can be contained, watching the leading signals, and accepting that a manageable failure may teach the organization more than another abstract warning.

The lesson is only useful if the prediction and decision survive. Otherwise the incident looks new, everyone repairs it under pressure, and the same trade is made again.

The risk was visible before the interruption

The company was onboarding more business and enterprise customers, ERP integrators, and warehouse integrations. Traffic grew every month. We kept it afloat with larger infrastructure, midnight deployments, and repeated performance work.

The warning signs were concrete:

• service interruptions were increasing;
• defects recurred in paths we had already repaired;
• work took four or five times its estimate;
• tests passed while defects reached production;
• developers spent time fighting the code and its tooling; and
• parts of the application resisted modern analysis and refactoring tools.

Any one signal can have another explanation. Estimates can be poor. A tool can be fashionable and unnecessary. A test can miss one unusual defect. Together, repeated over the same system while traffic rises, they describe a shrinking operating margin.

We used case-by-case prioritization and rotating capacity because a stable plan rarely survived contact with the next customer commitment. That mechanism helped the company respond. It also favored work with a visible sponsor over preventative work whose value appeared as an incident which did not happen.

Both kinds of work followed the evidence which was easiest to see.

A percentage cannot compare unlike risks

The obvious repair is a fixed allocation: reserve a fifth of the team for quality work and refuse to move it. That is better than an aspiration, but it still answers the wrong question when the risk changes.

Twenty percent may be excessive for a stable service with a generous capacity margin. It may be absurdly small when one more enterprise integration will overload a known path. Security remediation, latency headroom, test confidence, and dependency maintenance also do not become interchangeable because they share a backlog label.

The budget needs fields a decision can inspect:

This is a ledger, not a score. Collapsing an outage risk and a supply-chain risk into 17 only makes the disagreement harder to see. The fields force the premises into the conversation without pretending they share a unit.

Exceptions are part of the budget

A commercial priority may justify taking protected capacity. Losing a crucial partnership can be a larger risk than another week of fragile operation. The quality policy should make that decision possible while keeping both kinds of risk visible.

An exception should record four things:

1. the protected work being displaced;
2. the evidence-backed consequence now being accepted;
3. the reason the immediate business value outweighs it; and
4. the date or observation which reopens the allocation.

Engineering does not get to announce that a feature will definitely cause an outage when the evidence only shows reduced headroom. Commercial leadership does not need to understand the implementation before taking the risk seriously. Both need a description of likelihood, consequence, cost, and uncertainty which they can challenge.

Google’s example error-budget policy supplies one concrete version of this contract. When a service exceeds its budget over a stated window, most changes stop until the service returns within its objective. It also names exceptions, required postmortems, planning consequences, and an escalation owner. The useful part is not copying Google’s thresholds. It is agreeing on the response before the next urgent decision.

We did not have a rule strong enough to settle that trade automatically. We made the best case-by-case decision we could with commercial value on one side and technical forecasts of varying confidence on the other.

Connect the repair to the thing it enables

One decision stands out: we allocated people to features which would add work to already problematic performance paths before removing enough debt from those paths.

The alternative was not a broad modernization programme with no delivery date. It was to make the capacity repair a prerequisite of the feature which consumed that capacity. The commercial work and quality work belonged in one decision: we can add this load after this margin exists.

That framing is more useful than “engineering cleanup.” A prerequisite has a consumer, a reason, and a completion condition. If the feature is cancelled, the urgency can be reconsidered. If the feature doubles in value, the company can fund more capacity or accept more risk. The connection remains visible either way.

It also stops technical debt from becoming one infinite account. Some debt is ugly but cheap to carry. Some sits directly under the next revenue commitment. The second kind deserves capacity because of the change it enables, not because engineers dislike the code.

Make the next trade with better evidence

The organization and products in this account are deliberately anonymous. The effort ratios and outcomes are my recollection. Repository history can show active feature development, modernization, recurring fixes, and tooling work; it cannot prove what anyone understood in a private decision.

There was no single planning mechanism which ended the cycle. Complaints, interruptions, overruns, and repairs gradually made the system’s condition more legible. Technical work and customer delivery still competed afterward because that competition is part of running a company.

The improvement is not eliminating the conflict. It is making each repetition less surprising: retain the forecast, name its premises, connect repair to the business it enables, choose containment where possible, and compare the later outcome with what was predicted.

Technical leadership does not need to win every argument. It needs to make the important risks understandable enough that the company can choose which hills are worth defending—and recognize what happened when it chose differently.