Rollbacks that survive a schema migration

The rollback that does not work

The release goes out. Errors climb. Someone rolls back to the previous image, and errors get worse.

What happened: the new release included a migration that renamed a column. The new code expected the new name. The old code, now serving traffic again, expects the old one, which no longer exists. The rollback moved the application back in time; the database stayed in the present.

Point	Error rate (%)	Event
14:00	0.4
14:05	0.5
14:10	0.4
14:15	0.5	Deploy: v42 + rename
14:20	2.8
14:25	4.1	Alert: errors > 2%
14:30	3.9	Rollback: back to v41
14:35	9.6
14:40	11.2
14:45	10.8

Threshold: Alert at 2%.

Figure 1. The rollback that does not work: v42 renames a column, errors climb, and rolling back to v41 makes them worse because the old code expects the old name.Illustrative · sample data

An image rollback is fast and reliable precisely because it does so little: it points traffic at an image that already exists. That is also its limit. It does not touch data, schema, queues, caches or anything outside the container.

The rule

Every release must be compatible with the schema before and after its migration, and every migration must be compatible with the code before and after it.

If that holds, you can roll the code back at any point and the database will still make sense to it. The way to make it hold is to stop combining breaking schema changes and the code that depends on them in a single release.

Expand and contract

Split a breaking change into steps, each one safe to roll back.

Take renaming users.name to users.full_name:

Expand. Add full_name as a new nullable column. Old code ignores it. Release.
Dual write. Deploy code that writes to both columns and reads from name. Release.
Backfill. Copy existing values from name to full_name, in batches. No release needed.
Switch reads. Deploy code that reads from full_name and still writes both. Release.
Stop writing the old column. Deploy code that only uses full_name. Release.
Contract. Drop name in a later migration, once you are sure nothing (including rolled-back code) needs it.

Release 1
Expand, Live
Add full_name, nullable. Old code ignores it.
Release 2
Dual write, Live
helmsmanWrite both columns, read name.
No release
Backfill, In progress
bosunCopy name into full_name in batches.
Release 3
Switch reads, Live
helmsmanRead full_name, still write both.
Release 4
Stop old writes, Live
Only full_name is used.
Later
Contract, Queued
Drop name once nothing needs it.

Figure 2. Renaming one column as six small steps. Rolling back by one version at any point leaves a working application.

It is more releases. Each one is small, boring and reversible, which is the point. At every step, rolling the code back by one version leaves a working application.

Patterns for common changes

Adding a column: add it nullable or with a constant default. Make it required only after every writer sets it.
Removing a column: stop reading it, then stop writing it, release, and drop it later.
Renaming a column or table: expand and contract, as above. Never rename in place.
Changing a type: add a new column with the new type, dual write, backfill, switch, contract.
Adding a constraint: add it as not-validated if your database supports it, fix existing rows, then validate.
Adding an index on a large table: build it concurrently, in its own migration, outside the release.

Feature flags

Flags separate deploying code from enabling it. A new code path can ship turned off, be enabled for a subset of users, and be turned off again without a deploy.

For risky changes, a flag is often a faster and safer “rollback” than an image rollback:

Wrap the new behaviour in a flag, defaulting to off.
Deploy. Nothing changes for users.
Enable for internal users, then a percentage, then everyone.
If something goes wrong, turn the flag off.

Remove flags once the change is settled. Old flags are a source of confusion and bugs in their own right.

What a rollback cannot undo

Even with perfect schema discipline, some effects of a bad release are permanent:

Data written incorrectly. A bug that corrupted rows needs a data fix, sometimes a restore from point-in-time recovery.
Messages already sent. Emails, webhooks and notifications cannot be recalled.
Jobs already enqueued in a format the old code cannot read.
External state: payments taken, files deleted from storage, calls made to third-party APIs.
Cache entries in a new format that the old code chokes on.

Effect of a bad release	Image rollback	Flag off	Data fix or PITR
Code path serving traffic	Yes	Yes	No
Schema changesRestore rewinds everything, not one change	No	No	Partial
Rows written incorrectly	No	No	Yes
Jobs and cache entries in a new formatOld code must still read them	No	Partial	Partial
Emails, webhooks, paymentsAlready outside your system	No	No	No

Figure 3. What each kind of undo actually reaches. Only the code path is covered by an image rollback.

For each release, ask: if we roll back in ten minutes, what will still be different? Write the answer in the release notes.

Running migrations safely

Run migrations as their own step, before the release that needs them, not as part of container start-up.
Make migrations idempotent or tracked, so re-running the step does not apply changes twice.
Set a lock timeout so a migration that cannot get its lock fails quickly instead of queueing every other query behind it.
Rehearse anything non-trivial on a restored copy of production, with realistic data volumes.

How Kapten fits in

The Helmsman keeps previous images addressable and rolls back by re-pointing, behind the same health checks as a forward release.
A release plan lists the migrations it includes and flags those that are not backward compatible in the risk note.
The database migration rehearsal workflow has the Bosun restore a copy of production into an isolated environment, the Helmsman run the migration and the new release against it, and the Lookout report timing, locks and errors, all before you touch production.

Release and rehearsal capabilities are part of early access and confirmed during onboarding.

Checklist

This release works with the schema before and after its migration.
The previous release works with the schema after this migration.
No column or table is renamed or dropped in the same release that stops using it.
Large-table operations are concurrent or batched, and rehearsed.
Risky behaviour is behind a flag.
The release notes say what a rollback would not undo.

Built for your team