[After cutover] I wouldn't say the company hit the wall, but I would say we had major day to day challenges that needed to be solved quickly to avoid significant impact to the company. For example, our ontime ship, shipping on the date we commit to the customer, fell from 95% to about 75%, it was still not miserable but it was not good.
— Pete Solvik
The initial success of Cisco’s cutover to Oracle was, to say the least, something less than what was expected. Overall business performance plummeted as users attempted to deal with a new system that proved to be disturbingly unstable. On average, the system went down nearly once a day. The primary problem, as it turned out, was with the hardware architecture and sizing. Ordinarily correcting the deficiency would have required the purchase of additional hardware, thus increasing the total project expenditure. But Cisco had asked for, and gotten, an unusual contract from the hardware vendor. In their contract Cisco purchased equipment based on a promised capability rather than a specific configuration. As a result, the onus for fixing the hardware performance problems fell completely on the hardware vendor.
A second problem had to do with the ability of the software itself to handle the transaction
volume required in the Cisco environment. The design of the application exacerbated hardware
problems by inefficiently processing common tasks. In retrospect, it was clear where the company
had gone wrong in its final testing of the system. As Pond put it: “Some things were seriously
broken at big data volumes, . . . and we have a huge database. Our mistake was that we did not test
the system with a big enough database attached to it.” In testing the system, Cisco had run
individual processes sequentially rather than at the same time. In addition, only a partially loaded
database was used. After cutover, when all processes were running together over a fully converted database, the system lacked the capacity to process the required load.
The next two months were some of the most trying of the entire implementation. This was particularly true for the IT staff as it tried to grapple with the technical difficulties brought on by bringing the new system up. Fee described what it was like at this time:
It was tough, really stressful. This was a big thing, one of the top company initiatives.
There was a lot of focus on getting it done. We were working really long hours; making
decisions that would affect the company going forward We always knew we would make
it. It was always a “when,” not an “if.” There were [many] things you did not like about [the software].
ERP project status became the number one agenda item for weekly executive staff meetings. Strong vendor commitment from Oracle, the hardware vendor, and KPMG lead to an eventual stabilization of the software and improved performance. Solvik described the environment:
So for about 60 days we were in complete SWAT-team mode, get this thing turned around.
For example the president of the hardware vendor was our executive sponsor. This vendor
probably had 30 people on site at one point. They were all over it. They lost money on this big
time. It was great for them to get such a great reference, but it was a tough experience for
them. Remember we had bought a capability, so everything they did to add capacity was out
of their own pocket.