If the clock watcher announces that it is time to begin the back-out plan, you have to begin the back-out plan. This may happen if the upgrade is taking longer than expected or if it is complete but the tests continue to fail. The decision is driven entirely by the clock—it is not about you or the team. It’s can be disappointing and frustrating to back out of a complex upgrade but maintaining the integrity of the server is the priority.
Reverting the system back to its previous state should not be the only component of the back-out plan. Customers might agree that if only certain tests fail, they may be able to survive without that service for a day or two while it is repaired. Decide in advance the action plan for each potential failure.
After the back-out plan is executed, the services should be tested again. At this point it is important to record in your checklist the results of your changes. This is useful in reporting status back to management, record keeping for improving the process next time, or recalling what happened during a postmortem. Record specifics such as “implemented according to plan,” “implemented but exceeded change window,” “partial implemention; more work to be done,” “failed; change backed out,” “failed; service unusable; end of world predicted.” If possible, capture the output of the test suite and archive it along with the status information. This will help immensely in trying to remember what happened next week, month, or year.