A Characterization Test Is Not a Product Decision

A label setting behaved differently in test and production. Some customers had come to depend on the difference even though the product already let them choose the label size they wanted.

A characterization test could record both defaults. It could tell us which callers would notice if we made them equal. It could not answer the important question: should the difference survive?

Keeping it would protect customers who had never configured the setting. Removing it would give every environment one rule and make the customer’s choice explicit. The same observation supported arguments for either response. Only one response matched the product we wanted to operate.

Characterization is evidence for a compatibility decision. It is not the decision itself.

A passing test can preserve the wrong rule

The troublesome behavior looked small:

test environment        -> use one label size
production environment  -> use another label size
customer setting        -> override either default

The setting was already available. Customers who chose a size received it. The ambiguity belonged to customers who left it unset and relied on whatever their current environment happened to supply.

Changing a default then created three different statements which were easy to collapse into one:

1. Observed: some customers receive different results across environments.
2. Relied upon: changing either result will affect real workflows.
3. Promised: the product must preserve that difference.

A characterization test can establish the first statement. Usage data, support history, or a caller inventory may support the second. Neither proves the third.

Accidental behavior can acquire users. That makes it consequential, not automatically contractual.

Put observation and authority in separate artifacts

The test suite should describe what the system does before the change. The decision record should describe what the product will do after it.

For the label case, the observation table could have been this:

That table is useful because it exposes the affected group: customers without an explicit choice. It also shows that the product already has a route out of the ambiguity.

The decision needs a different table:

The first table belongs in tests. The second belongs with product and operational ownership. Asking a test to choose among those options quietly hands product authority to the first behavior somebody happened to record.

Compatibility deserves a reason, not a reflex

Preserving the old behavior is often the responsible choice. A public protocol, a stored format, an accounting rule, or a result consumed by uncontrolled clients may need a long deprecation window. A silent change can be more expensive than years of awkward implementation.

That was not the shape of this case. The behavior varied by environment, the customer preference was already configurable, and keeping the distinction made test results less representative of production. The compatibility mechanism would have preserved an absence of configuration rather than a declared preference.

We eventually required customers to configure the choice. There was no compatibility window. That was a sharper decision than I would use for an irreversible data format or an external API. It was defensible here because the remedy was available, local, and performed once.

The useful questions were not “does a test fail?” or “has anybody seen this behavior?” They were:

• Can affected users state the intended value explicitly?
• Can we identify those who have not?
• Is the old result a published promise or an implementation accident?
• What does preserving it cost every future change and investigation?
• What is the failure mode if we remove it?
• Can the decision be reversed without corrupting state or losing data?

Those questions turn compatibility from a moral instinct into a decision with scope.

Defaults are choices made on somebody else’s behalf

A default is convenient when one answer is safe for nearly everyone. It becomes dangerous when it guesses a preference that the system already knows how to ask for.

The label hierarchy should have been explicit:

customer choice
    ↓ if absent
documented system default
    ↓
same result in every equivalent environment

Instead, environment identity had become an invisible input. A customer could test one outcome and receive another after release without changing the request or configuration.

This is separate from the general rule that test and production should execute the same business logic. Even after agreeing on parity, somebody must decide which result becomes common. Copying production into test preserves one behavior. Copying test into production preserves another. Rejecting an unset choice replaces both. Parity identifies the inconsistency; product authority selects the rule.

Customer choice should outrank a system default. A system default should outrank an environment-specific guess. If no safe default exists, absence should be visible rather than translated into a preference the customer never made.

Existing usage changes the rollout, not necessarily the destination

Once people depend on a quirk, “just fix it” is incomplete advice. The dependency changes how the correction should be made even when it does not change the final policy.

There are several legitimate rollout shapes:

preserve
old behavior remains the contract

migrate
materialize the inferred value, notify affected users, then remove the branch

expire
keep the branch until a published date, then require explicit configuration

break deliberately
reject the ambiguous state now and provide a direct remedy

The label case took the last path. A larger or less identifiable population would have pushed toward migration or expiry. An external client with no configuration mechanism might have required preservation.

This is where characterization earns its keep. Tests and retained data help count affected states, enumerate observable results, and verify each rollout step. They make a deliberate break reviewable. They do not make it harmless.

Later retained artifacts from the same broad product area show why this distinction matters at scale. Tens of thousands of stored print preferences included stale identifiers and unsupported combinations. Correcting them required separate rules for remapping, deleting, and retaining values. Another change introduced an actor-specific default only for accounts created after a cutoff. These artifacts do not prove the earlier label decision or its outcome. They show that “keep what exists” stops being one policy as soon as stored state, account age, capability, and explicit preference disagree.

Write the new test after choosing the rule

Before the decision, characterization tests should pin the observed branches long enough to expose the change. After the decision, the suite should express the selected contract.

For an explicit-choice policy, the cases become:

configured value + test        -> configured value
configured value + production  -> configured value
missing value + test           -> configuration error
missing value + production     -> configuration error

The old environment-specific expectations should then be removed. Leaving them skipped or hidden behind a compatibility flag turns a completed decision back into permanent ambiguity.

The synthetic reproduction makes the authority split executable. The same observed-behavior suite accepts the old policy and rejects both candidate replacements. It cannot tell whether a common default or an explicit choice is preferable. Separate contract cases pass only after one policy is selected.

There is one more review question worth keeping: what evidence would reverse the decision? For this case, an inability to identify affected accounts, a device which could not accept the configured format, or irreversible document loss would have justified a compatibility period. Mere existence of the old branch would not.

A characterization test should make accidental behavior expensive to change silently. It should not make that behavior immortal. Record what exists, decide what deserves to survive, and let the final tests enforce the decision instead of pretending to make it.