Test the failover or you have not got one

A failover path that is not exercised stops working without anyone noticing. Sarah Wells, who ran the Financial Times' two-region platform, puts it flatly: an untested failover will turn out to be broken when it is needed, or will not fire automatically the way people assumed. It needs testing when introduced and regularly after.

Drift comes from ordinary change. Growth in load or a change in architecture can take traffic past what one region can carry alone, and only a test shows whether the capacity is still there. Documentation drifts as well, so the runbook needs checking on a schedule.

The FT practised failing over its API platform and website regularly, with a different engineer each time, so the real event felt familiar. A game-day checklist Wells borrows adds a sharper rule: leave out the people who always lead incidents, since they may not be around when it happens.

Rehearsals turn up The dependency you did not list, and they show what the system looks like with part of it down, which is where lag under load and lingering traffic can be observed.

After a test, the claim can carry a date. Before one, it is a description of intent.