A Surviving Mutant Is a Question, Not a Score

Two PHPUnit suites are green. The first checks a known product. The second also checks that a missing product throws ProductNotFound.

Run two mutations against them. The extra assertion kills the missing-exception defect, exactly as intended. Yet a 100% mutation-score gate rejects both suites. The stronger suite finishes at 50% because a second mutation changes no behavior for any input the method accepts.

The gate has calculated its number correctly. It has not interpreted the survivor.

Mutation testing earns its cost when it asks whether assertions notice a plausible defect. An escaped mutation therefore begins a review: did it expose a missing assertion, preserve the original behavior, reach code outside the test, or confuse the tool? Treating every survivor as an order to write another test replaces one vanity metric with another.

Two mutations ask different questions

The complete production target in the pinned experiment is a synthetic catalogue response translator:

final class CatalogueResponseTranslator
{
    public function priceCents(
        string $sku,
        int $status,
        array $payload,
    ): int {
        if ($status == 404) {
            throw ProductNotFound::forSku($sku);
        }

        return $payload['price_cents'];
    }
}

The loose comparison is deliberate. Infection runs against this one file with only two named mutators enabled.

Throw_ removes the throw keyword:

-throw ProductNotFound::forSku($sku);
+ProductNotFound::forSku($sku);

That changes the method’s caller-visible contract. A 404 can fall through and return the payload price rather than signalling an absent product.

EqualIdentical changes the comparison:

-if ($status == 404) {
+if ($status === 404) {

PHP defines == as equality after type juggling and === as equality with the same type. Here, the parameter is an int and 404 is an integer literal. Once a call has entered the method, both operands are integers. The two comparisons return the same Boolean value for every admitted status.

One mutation removes required behavior. The other expresses existing behavior more strictly. Infection can execute both changes, but only a reviewer can explain that difference.

The weak suite separates escaped from uncovered

The first suite has one example:

self::assertSame(
    10_000,
    $translator->priceCents(
        'desk-lamp',
        200,
        ['price_cents' => 10_000],
    ),
);

The test executes the comparison with status 200. Changing == to === does not change the result, so EqualIdentical escapes.

The test never enters the 404 branch. Infection reports Throw_ as not covered rather than escaped. That distinction matters. An escaped mutant ran against at least one covering test without being detected. A not-covered mutant did not get that chance.

The retained result is:

Mutation code coverage is 50%, while Mutation Score Indicator (MSI) and Covered Code MSI are both 0%. Those values are useful diagnostics. The missing-product branch has no test evidence, and the covered comparison mutation needs classification. Neither observation says “add any test which moves the percentage.”

One assertion kills the defect that matters

The strong suite keeps the known-product example and adds the missing-product contract from the preceding shared-contract experiment:

$this->expectException(ProductNotFound::class);

$translator->priceCents(
    'missing-sku',
    404,
    ['price_cents' => 0],
);

With throw removed, the method returns zero and PHPUnit fails because the expected exception never arrives. Infection records Throw_ as killed by the test framework.

The strict-comparison mutant still escapes. It receives both 200 and 404, but its answers remain identical to the original expression. The result becomes:

Mutation code coverage rises to 100%. MSI and Covered Code MSI become 50%. Nothing errored, timed out, produced a syntax error, or was skipped or ignored.

This is a genuine improvement. The suite now rejects the mutation which breaks the named failure contract. The remaining survivor does not identify another input partition to test. There is no integer value which makes the loose and strict comparisons disagree inside this method.

Classify the result before changing the suite

Infection’s official guide distinguishes killed, escaped, not-covered, errored, and timed-out mutants. Its machine-readable report gives the generated source, diff, category, and test output. That is enough to begin triage, not to finish it.

For each result, ask a different question:

These are reviewer classifications. A tool cannot infer the product requirement which makes a survivor meaningful. It also cannot prove equivalence for an arbitrary program. In this small method, the parameter type and generated diff make the equivalence argument inspectable.

That proof should remain narrow. Loose and strict equality are not generally interchangeable in PHP. Change the parameter to int|string, compare an untyped decoded value directly, or move the comparison before normalization, and the mutant may expose a real boundary. The conclusion belongs to this expression under this input contract.

A surviving equivalent mutant still deserves a code decision

Equivalent does not mean “ignore without looking.” The original == is weaker than the method’s actual invariant. Changing it to === would state that both operands are integers and prevent Infection from generating this particular mutation.

That can be worthwhile as an ordinary clarity improvement. It should not be described as a new test achievement. The behavior and assertions remain the same; the production expression now communicates the type constraint more directly.

If the original expression is clearer or cannot be changed, a narrow mutator exclusion can document the equivalent case. Broadly disabling equality mutations would throw away useful questions elsewhere. Adding a contrived test is worse: no valid input can kill this mutant, so the test must either exercise a different contract or pretend PHP admits a state the method has already excluded.

Sometimes the difficulty of killing a mutant exposes an overconstrained type, dead branch, or confused boundary. That is still useful. The response may be a code change, a contract correction, or a documented exclusion rather than another test method.

A threshold automates policy, not judgment

Infection provides --min-msi and --min-covered-msi for CI. When a result falls below either configured threshold, the command fails. The experiment deliberately sets both to 100%.

The weak run exits 1 at 0%. The strong run also exits 1 at 50%. Both failures are mechanically correct. The policy said every generated mutant must be killed; one was not.

The problem is the policy’s claim, not its implementation. Like a coverage target, a universal 100% gate assumes every generated mutant is distinguishable, relevant, and worth its maintenance cost. This example disproves that assumption without supplying a universal target.

A ratchet can still be useful after the existing survivors are classified. A changed-code gate can keep new work from weakening an understood baseline. A nightly wider run can explore code which would make every pull request too slow. A focused local command can answer one review question without turning the whole repository into a score campaign.

In each case, retain category counts alongside the percentage. A move from not covered to killed is informative even when an equivalent survivor leaves the headline score below target. A timeout becoming a kill means something different from an assertion becoming sensitive. One number hides those paths.

Spend a declared budget on a named risk

If the missing 404 example is already obvious from the contract, write it directly. A focused boundary test is cheaper than installing and operating a mutation tool. Mutation testing adds value when the team needs to check whether existing assertions distinguish selected code changes, or when a generated diff reveals a behavioral partition nobody named during ordinary review.

The experiment chose its budget before running:

• one production file;
• two mutators relevant to its comparison and exception;
• one worker;
• two ordinary suites which must pass first; and
• 2,000 milliseconds for each Infection command after the image is built.

The weak mutation run took 770 milliseconds. The strong run took 720 milliseconds. Both durations include disposable-container startup. They do not include the image build or dependency installation, and they say nothing about the expected cost in a Laravel application.

The small scope is a feature of the question. The experiment asks whether a missing-product assertion rejects a removed exception and how an equivalent survivor affects the score. Running every default mutator over a framework application would add expense without strengthening that answer.

Infection supports file arguments, one or many workers, source configuration, and detailed reports. Its command-line documentation also warns that parallel execution can produce misleading failures when tests share database or filesystem state. More workers are an optimization to measure, not a free multiplier.

Expand the budget when focused runs are stable and repeatedly find relevant survivors. Reverse the decision when triage and runtime cost more than direct boundary analysis, or when the important defect needs a database, queue, HTTP process, transaction, or deployment boundary which an in-process AST mutation does not supply. Choose the test boundary from the defect, not from the availability of a mutator.

Let the survivor earn the next change

Start mutation testing at one risk-bearing seam. Name the behavior which must survive refactoring and the plausible defect the assertions should reject. Select the relevant source and mutators, set a runtime budget, and keep every result category visible.

When a mutant survives, read its diff before touching the test suite. If it violates a requirement, add the smallest example which proves that requirement and rerun. If it preserves behavior for the valid input domain, document why and decide whether clearer production code or a narrow exclusion is warranted. If it was never covered, decide whether the current scope should own that path.

The useful outcome is not the highest percentage. It is a defensible answer to each generated question—and one more important wrong program that the suite can now reject.