A Release Gate Must Fail Before Incompatible Code Goes Live

Tue, 01 Nov 2016 00:00:00 +0000

A deployment completed, traffic reached the new code, and database queries started failing. The code expected columns or tables which did not exist.

The missing step was documented. After each manual deployment, somebody had to run a shell script which applied the pending migrations. The instructions were available, but the working knowledge remained concentrated in one operator. Other people followed the sequence for a while and eventually skipped it.

Running the migrations restored the application. The incident was not caused by an obscure database failure. A release had been allowed to report success before one of its prerequisites was true.

Deployments on Brian Faust

A Release Gate Must Fail Before Incompatible Code Goes Live