There are a couple of major scenarios we constantly come across. What I find fascinating is how incredibly different the scenarios turn out for the client.
For example, you would not believe how frequently a new client comes to us and they “know” they have good backups yet we find they have no backups at all (terrifying for anyone that has worked on business computer networks). They often are changing a backup tape or a hard drive out every day - religiously. They have a process they never miss. But surprise, surprise, nothing has been working as far back as we can see. Or it runs but fails and is worthless. Had their server gone down, they would have lost EVERYTHING. Gulp!
Another surprise is that we find the server has been reporting a major part failure (like a hard drive). The server may be under warranty but no one knew to check so it’s been hanging onto life by a thread.
So surprises are scary. I joke that our engineers should have a complete kit of tools, utilities, software as well as shots of tequila. But surprises still beat disasters.
We also get the covert calls. These are from firms that have another computer consultant but are in a crisis. They are dead in the water and their staff is twiddling their thumbs as they hemorrhage cash. The current firm can’t get them up or they have lost everything. They are panicking and trying to get our help covertly without upsetting the current firm. We always try to swoop in and save the day but sometimes it’s too late. Sometimes we can pull off a miracle. Sometimes we have to politely decline stepping into a nightmare.
But I love the midnight saves! We have a service where we monitor, patch, check on and proactively monitor our client's servers. Without getting into the technical details, we basically monitor thousands of things 24/7 365 days a year.
Every now and then something will be detected in the middle of the night and completely fixed before the first person walks in the door in the morning. It might be something simple like the email went down and is now back up. Or it could be major like a hard drive failed and the system almost crashed but it was rebuilt and tested. I find that very cool. The client generally has no idea how close they came to a disaster and we were able to proactively prevent it without them even seeing a single symptom.
Normally a situation like that could have been a catastrophe but instead it didn't even inconvenience them in any way! I wish we could brag about that but I suspect the client wouldn't “get” just how bad that would have been. So we brag to other IT guys who have been through the torture of a major firm outage before.
Servers and networks aren't like the building air conditioner or the copy machine. They need proactive maintenance and major things can be wrong and yet no one knows. They are often the most productive and valuable component in a business and yet taken for granted. They can make a ton of money or lose a ton of money yet they fly under the radar.
It’s interesting to see how identical problems can either be a big surprise, a total disaster or a midnight save.