In an effort to resolve fault-tolerance problems experienced with a third-party database, Google developed a database built on a fault-tolerant log using the Paxos replication algorithm. Several engineering problems encountered in building Paxos into a production-ready system are discussed, particularly involving testing and code maintenance