Rollback Cannot Restore a Contract You Already Removed

Release A stores a catalogue quote in amount_cents. Release B renames the column to amount_minor and deploys successfully. Every new B process is healthy.

One A process is still serving traffic during the rolling update. Its next query asks PostgreSQL for amount_cents and fails because the column no longer exists. Restoring the A image produces the same error. The old code returned; the contract it needs did not.

The same failure can survive in less visible places. An A queue worker can receive a B-only payload after the web rollout. An A process can read a B-only cache value after the application image is restored. Neither queued bytes nor cached bytes change because a deployment controller selected an older image.

A safe deployment is therefore a compatibility window. Every version which can still run must be able to read the state every version can still write. Rollback is one tested transition inside that window, not a universal undo button.

Compatibility belongs to readers and writers

“Backward compatible” is incomplete without four names:

• the reader;
• the writer;
• the state carrier; and
• the direction of the change.

The OPS-02 fixture uses three synthetic releases. A reads and writes amount_cents. B reads either field and temporarily writes both. C reads and writes only amount_minor after the old contract retires.

Those releases encounter four database phases:

B’s reader compatibility does not make B’s writer compatible with every phase. Before expansion, its new column is absent. After contraction, its old dual-write target is absent. The fixture retains 27 reader cells across the database, queue, and cache, plus 12 separate database-writer cells, because one “compatible” label would hide those directions.

The releases and quote are controlled examples, not claims about Brian’s systems. The fixture has no production traffic, state size, downtime budget, rollback objective, broker retention, cache behavior, or staffing model.

Expand before a new writer needs the field

The safe database sequence begins by adding amount_minor while retaining amount_cents. A still has the schema it expects. B can now be deployed as a reader which prefers the new value and falls back to the old one.

Only after that expansion does B become a dual writer. Its update changes both amount fields in one database statement. Temporary duplication is not the target model. It buys a period in which A and B can observe the same semantic quote while processes are replaced.

The destructive control does the opposite. It renames the column first. A’s read fails with PostgreSQL error 42703, and its post-rollback write fails with the same code. This is not a deployment-controller failure. The controller can restore an old image; it cannot infer or recreate the old schema and values.

Kubernetes documents that a rolling update gradually replaces old Pods. Its rollout status reports progress through process replacement. That establishes why old and new code may overlap. It does not prove that either version understands the other’s database rows, messages, or cache entries.

The PostgreSQL expansion in the fixture acquires an AccessExclusiveLock for the exercised ALTER TABLE. PostgreSQL’s ALTER TABLE documentation explains that lock level and distinguishes operations which scan or rewrite data from those which can update metadata. The local fixture has three rows. Its elapsed time says nothing useful about a production migration; the DDL form, table size, traffic, lock wait, and version must be evaluated in the real system.

A backfill is a restartable data process

Adding the column makes B executable. It does not make retained rows complete.

The fixture backfills three quotes in bounded batches. It persists the last completed quote ID. A simulated failure interrupts the run after quote-40; the next run resumes after that checkpoint and finishes at quote-42.

Checkpointing alone is not enough. A can update a quote between the backfill’s read and write. The fixture captures revision 1 of quote-42, then A changes the amount and advances the revision. A guarded backfill update expects the old revision and affects zero rows. Retrying from current state writes the new amount instead of restoring the stale snapshot.

Another control creates a disagreement deliberately: amount_cents is 1100 and amount_minor is 999. B does not silently prefer one. The migration records the row and applies the declared authority rule before continuing.

The order is visible:

expand schema
deploy compatible reader
enable dual-write
backfill with checkpoint and revision guard
reconcile disagreements
validate the new constraint

The new constraint begins as NOT VALID and becomes validated only after the rows satisfy it. PostgreSQL documents this split as a way to enforce new rows without performing the initial existing-row scan in the same operation. It is a database capability, not permission to skip the later validation gate.

The web rollout does not drain the queue

The fixture retains a v1 message produced by A before B deploys. Its delayed retry becomes available after the web release has advanced. B must still read amount_cents or translate that retained payload.

B also publishes an additive message containing both fields. A and B can read it during the overlap. C eventually publishes a v2-only message. A rejects that message with the named failure missing-amount_cents; C accepts it.

That rejection is terminal for the A consumer. Repeating the same incompatible attempt does not improve it. The retained repair action translates with B, preserves the message identity, and then retries under a compatible reader.

Laravel’s queue documentation describes workers as long-lived processes which do not notice code changes until restarted. Its deployment guidance separately requires queue workers, Octane, Reverb, and other long-running services to reload after deployment. Reloading the web fleet is not evidence that the old queue worker stopped, and stopping the worker is not evidence that its delayed payloads disappeared.

Before contraction, the fixture checks an owned process inventory for both web and queue workers. It then queries the durable queue for pending v1 payload dependencies. Both A process counts and the pending v1 dependency count must be zero. Healthy replica count alone is rejected as retirement evidence.

Cached values are deployed contracts too

A shared cache can keep the old shape alive after every process has restarted. It can also expose a new shape to old code after an image rollback.

The Redis fixture uses explicit envelopes:

{
  "schema": "additive",
  "quote_id": "quote-42",
  "quote_revision": 3,
  "amount_cents": 1234,
  "amount_minor": 1234,
  "writer_release": "B",
  "expires_at": 400
}

B reads an A-written v1 envelope. A and B read the additive envelope. A rejects a C-written v2-only envelope, while C reads it. An unknown v99 envelope and malformed JSON are classified as incompatible rather than disguised as cache misses.

There are two credible migration shapes. An additive envelope keeps one key while both readers understand the transition. Versioned keys keep v1 and v2 values separate while selection moves between them. Both require a retirement rule. A version suffix does not prove that no old reader can receive traffic.

The fixture removes only the owned v1 and additive quote keys and proves that an unrelated Redis sentinel remains. Laravel’s cache documentation warns that flush ignores the configured prefix. A global flush is therefore not a safe default for retiring one application’s value shape from a shared store.

Restoring A after B or C has written v2 does not delete that envelope. The recovery needs targeted invalidation, expiry, a compatible reader, or a rewrite from the authoritative source.

The rollback target changes during the deployment

“Roll back if it fails” should name both the artifact and the state phase.

The last safe A boundary in the fixture is before any release publishes a new-only artifact. B remains the rollback target through migration because it can read both shapes and operate the expanded schema. Once the old column is removed, dual-write B also fails with 42703. C succeeds because it writes only the new field.

If A must return after that point, the work is no longer an image selection. A repair release must restore the old schema and reconstruct its values, or adapt A to the state which now exists. Messages and cache values may still need translation or invalidation. Externally observed business effects may require compensation rather than either kind of rollback.

The contract gate therefore asks for evidence from each state carrier:

• no A web or queue process remains;
• no old-only write was observed for the declared window;
• the backfill checkpoint is complete and disagreement count is zero;
• the new constraint is validated;
• no pending v1 payload dependency remains;
• v1 cache values have expired or been removed from the owned key family; and
• every failure action, rollback target, and forward repair has an owner.

A single deployment-safety score cannot replace these facts. Compatibility, reversibility, duration, downtime, data volume, and operating cost can disagree with each other.

Compatibility logic is not always the cheapest answer

Expand-migrate-contract adds columns, fallbacks, dual writes, checkpoints, queries, translators, deployment stages, and cleanup. That cost is justified when uninterrupted mixed-version operation matters.

A stop-the-world deployment remains a credible alternative. For a small service with an accepted downtime window, bounded state, one owned worker set, and a tested restoration plan, one coordinated transition may be safer than temporary compatibility logic. This fixture cannot choose between them because it contains none of those production facts.

The reproduction retains 27 reader cells, 12 writer cells, three migrated PostgreSQL rows, five queue outcomes, seven cache outcomes, a 14-stage gate ledger, four score-free state-carrier decisions, 23 rejected counterfactuals, 11 hashed primary sources, and a clean export of commit f7f1b04035039aff012e8d160c3010059381c94b.

Before the next deployment, write the compatibility matrix rather than only the rollback command. Name every live reader, every live writer, the state each can leave behind, and the evidence which retires the old contract. Remove that contract only when the answer no longer depends on hope that the old code will never run again.