Datacenter to datacenter failover for many database backed websites is hard. For it to work well it must be designed into the app.. Please don't expect your Operations team to fire up DB replication and have it magically work.
- Autoincrement fields? DB supported, or hosted in the app, what happens if both sites are up? Can you get conflicting ids?
- Background jobs... how to you make sure only one copy is running or tolerate several?
- Latency... Can you even replicate the data in real time?
- Failover reponse... manual operation? Automated?
- IP/DNS issues, the simplest failover is to change DNS entries, but that can take a significant amount of time to replicate. Will your customers wait?
- How will you QA? Test failover in prod? Dual QA envs?
- Failback? How are you going to switch back?
Leave a comment