Test Cannot Prove Production When It Runs Different Rules

A label-layout change behaved correctly in test and disrupted document distribution in production. Both environments ran the same revision. The input looked equivalent. The test result was still incapable of predicting the production result.

The difference was deliberate. Test loaded one default and production loaded another. Users already had a way to select the layout they needed, but some had come to rely on the accidental production default. A new recommendation passed against the test rule while production continued through a different branch.

Nothing mysterious had happened between deployment and execution. We had called two different business decisions the same test.

An environment may change credentials, endpoints, capacity, and representative data. It should not quietly change the decision function whose behavior the test is supposed to prove.

The same revision is not the same program

Source control tells us which instructions were deployed. Runtime inputs decide which of those instructions execute.

Consider a small rule:

if environment is test:
    default layout = compact
else:
    default layout = standard

The revision is identical in both places. The effective program is not:

test input       -> test branch       -> compact
production input -> production branch -> standard

A successful test has proved only the first path. It says nothing about the second unless another test executes that path with production-equivalent inputs.

This is why “works in test” and “same commit is deployed” can both be true without contradiction. Neither establishes behavioral parity.

Separate deployment configuration from business choice

Environments genuinely need different values:

database address
credentials
queue capacity
log destination
remote sandbox endpoint
feature rollout percentage

Those values connect and operate the application. They do not necessarily change the meaning of a booking, invoice, or label request.

Business choices look different:

which label layout wins
which tax rule applies
whether an order is eligible
how a shipment is routed
when a payment is considered settled

Putting the second group behind an environment check does more than configure a deployment. It creates an environment-specific product.

That can be intentional. A destructive payment call should reach a provider’s sandbox from test. A staged rollout may expose new behavior to a bounded audience. A simulator may need controlled failure modes which production does not. In each case the difference should be named, observable, and exercised as a difference.

The dangerous form is an unnamed accommodation:

production needs this odd default for old users
test needs the new default so current work can proceed

Now test is no longer a detached representation of production behavior. It is a second policy implementation.

Defaults are decisions with invisible callers

An explicit setting has an owner. A default acquires callers without recording them.

Suppose layout selection appears to work like this:

explicit request
stored user preference
system default

If a user sends neither an override nor a stored preference, the system default becomes part of that user’s effective contract. Changing it can therefore change output for every silent caller.

The problem is not that defaults exist. A system needs a final answer when omission is valid. The problem is pretending that a widely observed default is merely internal because nobody selected it explicitly.

The label case had an additional warning sign: users could configure the required layout but some had not. Environment-specific defaults absorbed that ambiguity for a while. Each exception made the result harder to predict:

same request
same stored data
same revision
different environment
different document

Supporting that state indefinitely would make every later recommendation, carrier change, and regression test negotiate two implicit products.

One resolver should own precedence

The stronger design is a decision function whose inputs are business facts:

resolve(
    requested layout,
    stored preference,
    allowed layouts,
    driver default
)

Its precedence can be explicit:

1. use the requested layout when it is allowed;
2. otherwise use the stored preference when it is allowed;
3. otherwise use the driver’s declared default.

Environment is not an input.

A retained implementation follows that shape. Its tests cover a request override, a stored preference, disallowed values, and a driver fallback. The resolver reads the allowed layouts for the selected service, but it does not ask whether it is running in test or production.

That does not prove every surrounding system is identical. Databases, provider sandboxes, fonts, printers, and customer data can still differ. It proves a narrower and valuable invariant: given the same resolver inputs, the layout decision does not change because of the deployment name.

The DBG-10 fixture makes that invariant executable. It runs one synthetic decision matrix under development, test, and production labels and requires identical results. A mutation adds one production-only default and the same matrix immediately disagrees.

Allowed values are policy, not an escape hatch

A user preference should not necessarily beat every other input. A selected layout may be invalid for the chosen service or unsupported by the downstream system.

That gives the resolver two independent jobs:

precedence: which source should win?
validity:   may this value be used here?

Conflating them produces another kind of accidental default. If the explicit request is invalid, blindly accepting it can fail downstream. If the resolver silently replaces it, the user may receive a document they did not ask for.

The retained resolver treats an allowed explicit request as highest priority, then an allowed stored preference, then the driver fallback. A nullable form can also report that neither explicit nor stored input is usable before the caller applies its fallback.

The precise failure behavior is a product choice. An API may reject an unsupported explicit layout rather than fall back. A user interface may prevent the selection. A legacy import may need a documented compatibility mapping. The parity rule does not dictate which choice is correct. It requires every environment to execute the same choice for the same facts.

Representative data still matters

Removing environment branches does not make test equivalent to production.

Production may contain stored preferences which test never seeds. It may have old rows with values the current interface no longer creates. It may receive a layout omitted by ordinary fixtures. This series has already covered failures which live in data rather than code.

The two problems need different repairs:

Calling all four “environment drift” hides the mechanism. Debugging improves when the mismatch can be named.

Compatibility has a cost and an owner

There was an alternative to removing the old label accommodation immediately: retain the production default, identify every user relying on it, provide a migration window, record explicit preferences on their behalf with approval, and remove the branch after adoption.

That approach reduces surprise. It also preserves two behaviors while the migration runs, requires a reliable way to identify silent reliance, and keeps new work coupled to the old exception.

In the supplied case, the accommodation was removed without a compatibility window. Users were required to configure the layout they actually wanted. That decision traded short-term support friction for one explicit rule.

It is not automatically the right migration policy. The decision should depend on blast radius, contractual promises, the cost of incorrect output, ability to identify affected users, and the support capacity available. A high-impact financial rule deserves a different rollout from a preference which is already self-service.

What should not survive is an unowned environment branch with no removal condition. If compatibility is necessary, give it:

affected population
old and new behavior
selection mechanism
observation
owner
deadline
removal test

Then it is a migration, not folklore.

Verify parity as a matrix

The smallest useful parity check feeds the same cases to the same resolver under every supported deployment label:

The environment label is deliberately absent from expected behavior.

This kind of test will not catch a database loaded with different preferences or a provider sandbox which renders documents differently. It catches the specific regression at issue: somebody makes a business result conditional on the deployment name.

Architecture checks can help locate direct environment reads in domain code, but source shape is only a warning. A feature-assignment service may read deployment configuration legitimately while exposing an explicit variant to the domain. Conversely, a value loaded indirectly from separate environment files can change behavior without any obvious if production branch.

The behavioral matrix is the proof. Static checks only narrow the search.

A pre-production environment should be allowed to disagree honestly

Perfect replicas are usually impractical. Test may use smaller infrastructure, sanitized data, fake credentials, and provider sandboxes. Those differences are useful when they are visible.

For each material difference, record:

what differs
why it differs
which claim test can no longer make
how production behavior is verified safely
who owns removal or continued acceptance

If the remote sandbox generates a placeholder document, test can still prove request mapping and error handling. It cannot prove the real document’s dimensions. If test has less traffic, it can prove functional selection. It cannot prove production capacity.

This language is more honest than calling an environment “production-like” and letting the adjective absorb every unknown.

The label failure was not evidence that test environments are useless. It was evidence that we had asked one product rule to answer differently depending on where it ran, then treated one answer as proof of the other.

Keep deployment concerns configurable. Keep business inputs explicit. Run the same decision function everywhere. When an environment must differ, name the mechanism and subtract the claim it can no longer prove.