Overnight Network Recovery

Critical system failures can happen at any time. If we need to work through the night to fix a problem, we’ll do it.

Situation

Toronto-based imason is a leading SharePoint developer, technology consultancy and software development company.

At around 7:30PM on a Monday night, Quartet received an alert that imason’s firewall server failed to boot after updating. A quick look at the error messages and server specs told us that the problem was most likely hardware related, and would require new equipment.

A defunct firewall server means no ingoing or outgoing email, no software updates, no company website and no access to the internet; imason would have been cut off. This problem had to be fixed before 9:00AM the following Tuesday morning.

Challenges

Coordinating Quartet Tier 1,2 and 3 technical resources after hours on a complicated, critical issue. Activating manufacturer supply chain after hours and installing, testing and integrating the best solution ASAP.

Solution

Seven Quartet staff were involved in the solution. In the span of three hours, over 30 internal email exchanges took place and two solutions were tested.

Like most IT issues, testing and examination are necessary to pinpoint the real problem. After working for an hour with the hardware vendor, trying to flash the adapter and reseat the cards, nothing was working. The most probable cause was a failed card, but there was no way to be sure.

Regardless, Quartet ordered a new card from the vendor’s Toronto warehouse on priority delivery. It arrived 2 hours later. In the meantime, the rest of the Quartet team was compiling a list of different scenarios in the event that the new card did not fix the problem.

By 10:00 that night, our team had a list of 3 possible scenarios to reboot imason’s server, and a game plan for each. If the new card failed, they would be ready.

Just before midnight, Quartet arrived at imason’s servers and installed the card. The system booted successfully and after half an hour of more testing, the server was given a green light. The rest of the team was notified and taken off standby. Problem solved.

Results

“Awesome job recovering from what could have been a disastrous Tuesday morning. Without that firewall we are dead to the world. Thank you for staying up half the night to get us back on line.”

- Jeff Dunmall, co CEO, iMason inc., 9:23 AM