Test-Design on Brian Faust

A Characterization Test Is a Boundary, Not a Recording

Mon, 01 Apr 2024 00:00:00 +0000

An undocumented postage calculator needs to be split apart. Before touching it, we put its current behavior under test.

One suite stores the returned quote, every intermediate calculation, the order of the diagnostic fields, and the sequence in which four pricing rules run. Another asserts the caller’s result across weight, insurance, destination, and discount boundaries.

Both suites are green. Both characterize the current implementation. They do not buy the same freedom to change it.

A Surviving Mutant Is a Question, Not a Score

Fri, 01 Mar 2024 00:00:00 +0000

Two PHPUnit suites are green. The first checks a known product. The second also checks that a missing product throws ProductNotFound.

Run two mutations against them. The extra assertion kills the missing-exception defect, exactly as intended. Yet a 100% mutation-score gate rejects both suites. The stronger suite finishes at 50% because a second mutation changes no behavior for any input the method accepts.

The gate has calculated its number correctly. It has not interpreted the survivor.

An Interface Does Not Make a Fake Substitutable

Thu, 01 Feb 2024 00:00:00 +0000

An HTTP product catalogue and an in-memory fake both implement this interface:

interface ProductCatalogue
{
 /** @throws ProductNotFound */
 public function priceCents(string $sku): int;
}

Both return 10000 for CHAIR-BLACK. Both make the checkout test green. Both pass PHP’s signature checks.

Now seed one divergence. When the product is absent, the HTTP adapter throws ProductNotFound while the fake returns zero. The interface remains satisfied, and the familiar checkout test still passes because it never asks for a missing SKU.

A Mock Fortress Tests a Call Script

Mon, 01 Jan 2024 00:00:00 +0000

Two tests produce the same checkout quote. One builds five Mockery doubles, specifies every argument, fixes their global call order, and makes a serializer mock return the expected JSON. The other resolves the use case through Laravel’s container, replaces one HTTP catalogue, and asserts the JSON emitted by the real policies and serializer.

Both are green. That tells us they agree with today’s code. It does not tell us what either test will reject tomorrow.

Choose the Test Boundary from the Defect

Fri, 01 Dec 2023 00:00:00 +0000

Four test boundaries protect the same reservation flow. Then four defects arrive:

a quantity of zero becomes valid;
a database migration loses a uniqueness constraint;
an inventory provider renames available to stock; and
browser JavaScript submits an empty string instead of the entered quantity.

Calling all four tests “feature tests” would tell us almost nothing. Putting them into a pyramid would tell us how many tests we hope to own, but not which one can catch each defect.

Coverage Tells You What Ran, Not What Your Tests Would Catch

Wed, 01 Nov 2023 00:00:00 +0000

Here are two PHPUnit suites for the same shipping-fee rule. Both report 100% line coverage. Both also report 100% branch and path coverage. One suite lets an off-by-one defect escape; the other catches it.

Nothing is wrong with the coverage report. The mistake is asking it a question it cannot answer.

Coverage can tell us which executable structure a test run visited. It cannot tell us whether the chosen inputs and assertions would notice a defect that matters. The distinction sounds modest. It changes how a coverage target should be used.

Test Process State, Not Message Choreography

Sun, 01 Oct 2023 00:00:00 +0000

A strict mock test expects five calls in order:

load process
check message ID
dispatch ReserveInventory
save process
record inbox

Every expectation passes. Unfortunately, save process writes the old awaiting_payment state. The command says inventory should be reserved while the durable process still says it is waiting for payment.

Now fix the persistence and move record inbox before save process inside the same database transaction. The committed process, inbox, and outbox rows are identical. The strict test fails because two calls changed places.