Here are two PHPUnit suites for the same shipping-fee rule. Both report 100% line coverage. Both also report 100% branch and path coverage. One suite lets an off-by-one defect escape; the other catches it.
Nothing is wrong with the coverage report. The mistake is asking it a question it cannot answer.
Coverage can tell us which executable structure a test run visited. It cannot tell us whether the chosen inputs and assertions would notice a defect that matters. The distinction sounds modest. It changes how a coverage target should be used.
Four paths can still miss the important value
The complete production code in the experiment is one method:
final class ShippingFee
{
public function cents(int $subtotalCents, bool $priority): int
{
$base = $subtotalCents >= 5_000 ? 0 : 500;
return $base + ($priority ? 1_000 : 0);
}
}Orders at or above 5,000 cents receive free standard shipping. Priority shipping adds 1,000 cents either way.
The first suite exercises all four control-flow combinations:
| Subtotal | Priority | Expected fee |
|---|---|---|
| 4,999 | no | 500 |
| 5,001 | no | 0 |
| 4,999 | yes | 1,500 |
| 5,001 | yes | 1,000 |
Every line runs. Both outcomes of each conditional run. The four possible paths through the two conditionals run. PHPUnit, with Xdebug’s branch-check mode, reports exactly that:
Classes: 100.00% (1/1)
Methods: 100.00% (1/1)
Paths: 100.00% (4/4)
Branches: 100.00% (7/7)
Lines: 100.00% (2/2)This is useful information. There is no unvisited branch hiding in the
method. But no example uses the value 5_000, where the business rule changes
state.
Control-flow coverage groups 5_000 and 5_001 together. Both make the
condition true. The report records the branch they take, not every meaningful
value inside the set that takes it. In this rule, the exact boundary is where
>= earns its extra character.
Change one character and the score stays perfect
Mutation testing gives us a direct way to test whether the assertions notice that character. Infection can replace the comparison with this one:
-$base = $subtotalCents >= 5_000 ? 0 : 500;
+$base = $subtotalCents > 5_000 ? 0 : 500;
This is not a random syntax disturbance. It models a familiar boundary defect: an order at exactly 5,000 cents now pays 500 cents for standard shipping.
The four tests still pass. Their inputs are 4,999 and 5,001, and the mutation does not change either result. Infection records one covered mutant and one escaped mutant. Mutation code coverage is 100%; Covered Code MSI is 0%.
Those two percentages answer different questions. The first says the mutated code was executed. The second says the tests did not distinguish the original program from this particular defective version. Execution was complete; discrimination was not.
One boundary example changes the evidence, not the coverage
The second suite keeps the four original cases and adds one:
| Subtotal | Priority | Expected fee |
|---|---|---|
| 5,000 | no | 0 |
The structural coverage report does not move. It remains 100% for the same two lines, seven branches, four paths, one method, and one class. The new example travels a path that the suite had already visited.
The mutation result does move. With > in place of >=, the new assertion
receives 500 instead of 0 and fails. Infection reports the same mutant as
killed and Covered Code MSI becomes 100%.
The complete pinned reproduction retains both test suites, coverage reports, mutation logs, commands, tool versions, durations, and exit codes. It uses PHP 8.4.12, Xdebug 3.4.5, PHPUnit 12.5.31, and Infection 0.34.0. The result is deliberately small: one method, one mutant, and one run. It demonstrates a mechanism, not a universal relationship between coverage and test quality.
Coverage is an inventory of exercised structure
PHPUnit’s coverage documentation distinguishes line, branch, and path coverage. Line coverage asks whether each executable line ran. Branch coverage asks whether each branch ran. Path coverage asks whether each possible path through a method ran. Xdebug’s coverage mode supplies the branch and path data used by this experiment.
Each stronger metric closes a real blind spot in the weaker one. A line can run without both branches running. Every branch can run while some combination of branches remains untested. Moving from line to branch to path coverage can therefore reveal missing structural exercise.
None of the three metrics records the business partition between 4,999, 5,000, and 5,001. That partition comes from the rule, not from the control-flow graph.
This makes coverage valuable as an inventory signal:
- uncovered lines identify code for which the current run supplies no execution evidence;
- uncovered branches and paths expose structural cases hidden by line coverage;
- changes in the report can reveal an accidental loss of exercised surface; and
- file-level results can direct review toward neglected areas.
The word inventory sets the boundary. Finding stock on a shelf does not tell us whether it works. Running a line does not tell us whether an assertion would reject an important wrong result.
Mutation testing adds a question, not a better vanity score
It would be easy to replace one target with another: stop chasing coverage and chase Mutation Score Indicator instead. That repeats the original error with a more expensive number.
Mutation testing can only ask about the mutations a tool generates. Some mutants are equivalent to the original program. Some represent defects nobody cares about. Missing requirements do not appear merely because a mutator is clever. The Infection documentation also distinguishes mutation code coverage from the share of covered mutants the tests kill; neither metric decides which behavior deserves protection.
There is a cost as well. Infection runs tests against changed versions of the program. On a large suite, exhaustive mutation can consume enough time that a focused local run, a nightly budget, or a changed-code policy is more useful than a mandatory full run. The right budget depends on runtime and mutant relevance, not on the appeal of 100%.
For the shipping rule, a targeted boundary mutator earns its cost because it models the exact defect under discussion. If the risk were serialization, transaction isolation, or compatibility with another implementation, a mutation score would be weaker evidence than an integration or contract test. The defect and boundary should choose the test technique.
Start with the defect the suite must reject
A coverage report is most helpful after its job is stated precisely. Use it to find structure the suite never exercises. Do not make the percentage stand in for the behavior the suite must protect.
For a threshold rule, write down the partitions before reading the score: below the threshold, exactly at it, and above it. For a substitute implementation, state the shared behavioral contract and run the same contract tests against each implementation. For a database boundary, name the query, constraint, or transaction failure that an isolated unit test cannot expose.
Then use stronger evidence where the risk justifies it. A focused mutation can check whether an assertion distinguishes a plausible code defect. An integration test can exercise a real framework boundary. A contract suite can make substitutability measurable.
Coverage still belongs in that workflow. It answers, with increasing structural detail, “what did this run execute?” The shipping experiment shows why the next question has to remain separate: “which important wrong programs would these assertions reject?”