A Release Gate Must Fail Before Incompatible Code Goes Live

A deployment completed, traffic reached the new code, and database queries started failing. The code expected columns or tables which did not exist.

The missing step was documented. After each manual deployment, somebody had to run a shell script which applied the pending migrations. The instructions were available, but the working knowledge remained concentrated in one operator. Other people followed the sequence for a while and eventually skipped it.

Running the migrations restored the application. The incident was not caused by an obscure database failure. A release had been allowed to report success before one of its prerequisites was true.

Repeated deployment steps should not depend on memory. More importantly, a release must not activate incompatible code until it can observe that the required schema and data state exists.

A checklist records intent; a gate controls state

Written instructions can explain why a step exists, how to inspect it, and what to do when it fails. They cannot stop a process which never reads them.

The legacy deployment effectively said:

pull new code
restart application
remember to run post-deploy script

The system could reach “deployed” after the second line. The third line was a request to a person, not a condition of activation.

A release gate reverses the relationship:

prepare release
run required transitions
verify required state
activate release only if verification passes

If the transition fails, the release remains inactive. If verification cannot run, the release state is unknown rather than successful.

That distinction survives changes in tooling. A managed deployment service, a shell script, a container rollout, and a hand-built release process can all enforce the same state machine. Automation helps because it executes consistently; the invariant matters more than the brand of automation.

Name the dependency between code and state

“Run migrations” is too broad to describe the release risk. A useful plan says which code requires which state:

new reader requires column shipment.status_reason
new writer requires table booking_attempts
background operation backfills existing rows
old reader does not understand the new enum value

The dependency determines safe ordering.

If new code reads a new nullable column but old code ignores it, the schema can often arrive before activation. If a migration removes a column still read by the old release, it cannot safely run while old processes remain. If a backfill touches millions of rows, coupling it to the web deployment may exceed the release window even though the final schema is required later.

The gate should therefore ask a concrete question:

Is every prerequisite for the code being activated complete and compatible
with every code version which may still be running?

That is stricter than “did the migration command exit zero?” A successful command may have applied only part of a multi-stage transition, or the release may require a data operation which is deliberately not a schema migration.

Zero downtime creates an overlap contract

A zero-downtime deployment normally means old and new processes overlap. The release directory may switch atomically, but workers, in-flight requests, and several application nodes can continue executing the previous code.

Schema changes must tolerate that overlap:

expand:
    add state understood safely by old code

migrate:
    write or backfill both representations as needed

switch:
    activate code which uses the new representation

contract:
    remove old state only after old code can no longer execute

Not every change needs four releases. The model exists to expose when one change cannot be safely compressed into one activation.

For the original missing-migration incident, the immediate invariant was simpler: the required schema had to exist before the new code served traffic. Moving to a managed zero-downtime process prevented the remembered script from being omitted. That outcome does not mean every migration should always run inside every deployment.

Automatic is a policy, not a permanent answer

A retained deployment history shows the migration step moving between enabled and disabled states. Data operations were similarly enabled, paused for release windows which required separate verification, and enabled again. The current path leaves schema migrations separate while processing approved application operations synchronously.

That history is not evidence of indecision. It exposes a real operational trade-off:

automatic transition
  advantage: omission blocks the release instead of reaching production
  risk: a slow, unsafe, or irreversible transition sits on the activation path

separate transition
  advantage: dangerous work gets its own timing, observation, and approval
  risk: activation can outrun it unless an explicit gate checks completion

The bad state is neither “automatic” nor “manual.” It is separation without a machine-observable dependency.

If migrations are intentionally disabled in the deploy script, the release needs another source of truth:

required transition set
completed transition set
compatibility state
approval or ownership
activation decision

A comment beside a disabled command explains local intent. It does not prove that another process completed the work.

Fail fast before side effects accumulate

The successor deployment script uses fail-fast shell behavior. If a required command exits unsuccessfully, later steps do not continue as though nothing happened.

That control matters because deployment steps have side effects:

install dependencies
run transitions
change active release
reload processes
restart workers
clear or warm caches

Continuing after an early failure can combine an old dependency set, new code, partial schema, and restarted workers into a state no release plan described.

Fail-fast behavior is necessary but incomplete. Shell commands can return zero while leaving the wrong state, and some tools treat warnings or skipped work as success. Follow consequential steps with state verification:

command: apply required schema transitions
verify:  required transition identifiers are recorded

command: run approved data operation
verify:  operation reached terminal success

command: activate release
verify:  health check exercises a dependency on the new state

The check should observe the contract, not merely the command line which was supposed to create it.

Concurrency needs one transition owner

When several application nodes deploy together, all of them may try to apply the same migration. Some database operations are naturally idempotent; many are not safe to race.

Current Laravel documentation describes migrate --isolated as using an atomic lock so only one server runs migrations, provided the servers share a supported cache backend.

That solves concurrent ownership for the command. It does not make the migration compatible, fast, reversible, or safe under load. Nor does it prove that every node waits for the winning process before activating dependent code.

The release plan still needs:

one transition owner
all activations wait for the result
failure is visible to every participant
lock loss and timeout have defined outcomes
the resulting state is verified

An isolated command is a mechanism inside that plan, not the plan itself.

A release needs more states than success and failure

The DBG-11 fixture models three plans:

automatic:
    transition -> verify -> activate

separated:
    transition completed elsewhere
    verify recorded evidence -> activate

unsafe:
    transition omitted -> activate

The automatic plan activates because its prerequisite becomes observable. The separated plan activates only after fixture evidence names the completed transition and compatible overlap. The unsafe plan is rejected before activation.

This small model uses release states which operational tools often blur:

prepared
blocked
transitioning
ready
active
failed

“Blocked” is not failure. It says the release has not earned activation yet. That is safer than treating a missing decision as success.

The fixture does not run a database, a web server, Laravel, or a deployment platform. It cannot prove migration duration, locking, rollback, or production compatibility. It proves only the declared state transitions and the rejection of missing prerequisite evidence.

Recovery should not depend on the original operator

During the supplied incident, the recovery was direct: run the omitted migrations, then the application could query the expected schema. That was possible because the failure mechanism was recognized.

A useful recovery record should let another operator reach the same decision:

This is where a runbook earns its place. It begins with an observable symptom and leads to a bounded decision. It does not ask the reader to remember which private script one person usually runs.

The original instructions were not worthless. They were assigned too much authority. Documentation should explain and support the release control, not stand in for it.

Make exceptions expire

There are good reasons to remove a transition from the ordinary deployment path:

• it cannot complete within the release window;
• it carries destructive or irreversible risk;
• it needs a backup, replica check, or maintenance window;
• it must be observed before application activation;
• it changes data semantics which require business approval; or
• it needs staged backfill and compatibility releases.

Each exception should name the affected release, owner, prerequisite evidence, activation rule, recovery plan, and removal condition. Without those fields, “temporarily disabled” becomes an unbounded second deployment process.

The retained history is useful precisely because it prevents a simplistic lesson. Automatic migrations prevented a repeated human omission. Later, separating some transitions was rational when their risk needed independent verification. The invariant remained the same: incompatible code must not become active merely because the deployment command reached its final line.

A release is ready when its dependencies are true, not when somebody has finished typing. Put recurring work in the path, stop on failure, verify the result, and require explicit evidence whenever a risky transition moves outside that path.