The affected medical facilities immediately implemented
their local contingency plans, which consist of three levels:
the first of those is a fail-over from the Sacramento Data
Center to the Denver Data Center, according to Bryan D.
Volpp, associate chief of staff and clinical informatics. Volpp
assumed that the data center in Sacramento would move
into the first level of backup—switching over to the Denver
data center. It didn’t happen.
On that day, the Denver site wasn’t touched by the out-
age at all. The 11 sites running in that region maintained
their normal operations throughout the day. So why didn’t
Raffin’s team make the decision to fail over to Denver?
“What the team in Sacramento wanted to avoid was putting
at risk the remaining 11 sites in the Denver environment,facilities that were still operating with no glitches. The
problem could have been software-related,” Raffin says. In
that case, the problem may have spread to the VA’s Denver
facilities, as well. Since the Sacramento group couldn’t pin-
point the problem, they made a decision not to fail over.
Greg Schulz, senior analyst at The Storage I/O Group,
said the main vulnerability with mirroring is exactly what
Raffin feared. “If I corrupt my primary copy, then my mirror
is corrupted. If I have a copy in St. Louis and a copy in Chi-
cago and they’re replicating in real time, they’re both cor-
rupted, they’re both deleted.” That’s why a point-in-time
copy is necessary, Schulz continued. “I have everything I
need to get back to that known state.”
According to Volpp, “the disruption severely interfered
with our normal operation, particularly with inpatient and
outpatient care and pharmacy.” The lack of electronic
records prevented residents on their rounds from accessing
patient charts to review the prior day’s results or add orders.
Nurses couldn’t hand off from one shift to another through
Vista, as they were accustomed. Discharges had to be writ-
ten out by hand, so patients didn’t receive the normal lists of
instructions or medications, which were usually produced
electronically.