Rollback Cannot Restore a Contract You Already Removed

Wed, 01 Oct 2025 00:00:00 +0000

Release A stores a catalogue quote in amount_cents. Release B renames the column to amount_minor and deploys successfully. Every new B process is healthy.

One A process is still serving traffic during the rolling update. Its next query asks PostgreSQL for amount_cents and fails because the column no longer exists. Restoring the A image produces the same error. The old code returned; the contract it needs did not.

The same failure can survive in less visible places. An A queue worker can receive a B-only payload after the web rollout. An A process can read a B-only cache value after the application image is restored. Neither queued bytes nor cached bytes change because a deployment controller selected an older image.

Observability Must Answer the Next Operator Question

Mon, 01 Sep 2025 00:00:00 +0000

A catalogue quote request returns 503 after its price dependency times out. One telemetry bundle produces five uncorrelated log lines, a metric labelled with request ID and raw URL, and three pages. The pages do not name an owner, affected scope, hold duration, or first query.

The other bundle produces fewer records and one page. It says which route, region, and release are affected. Its first query measures the user-visible failure. An exemplar selects a failed trace, the trace locates the dependency wait, and a correlated event names the application failure class. The runbook asks whether bypassing that dependency is safe.

Observability and Deployment on Brian Faust

Rollback Cannot Restore a Contract You Already Removed

Observability Must Answer the Next Operator Question