What Happened?
April 15th, 2010 by ComputerBob
This Web site’s current hosting provider has been very reliable for the past few years that I’ve used it.
I know, because I have two different site-uptime-checking services that check the availability of my Web sites every 30 minutes, 24 hours a day and send me email notices if any of my sites are down.
Then at the end of every month, each of those two services sends me a monthly uptime report on my sites.
For most months in the past 3 to 4 years, this Web site has had 100% reliability, has never had less than 99.9% reliablility in any month, and has never had any unscheduled downtimes of more than about 20 minutes.
Until yesterday morning.
My Web host runs more than 50 Web servers, but apparently, yesterday morning, cybercriminals breached the security of only the server that hosts all of my Web sites. As a result, my Web host decided that the most secure course of action was to immediately take that server offline, erase its hard drives, completely rebuild it from scratch, and then completely restore its thousands of Web sites, databases and config files from backups.
To make matters even worse, hardware issues surfaced partway through the rebuilding process, so they had to replace the entire server with a brand new server — and then start the whole rebuilding-from-scratch process all over again.
During that lengthy process, this site was completely unreachable for about 24 hours.
In fact, the reason that this site is still running really slowly right now, and will probably continue to do so for the next several hours, is because most of the new server’s CPU and bandwidth is currently being used to transfer and restore the last 60 GB of data from the old server.
And my email won’t start working again until later on, after the server’s email password files have been restored.
So now you know the rest of the story.
P.S. Despite — and partly because of — the catastrophic incident of the past two days, I am still very happy with my Web hosting provider because I’m sure that they’ve locked down the new server to be even more secure than the old server was; I anticipate that, once they’re done restoring everything to the new server, this site even faster than it used to be; and they did a pretty good job of keeping me informed of the status of the server rebuild and restoration process — with one major exception: Early this morning, before I left for work, a Level 1 Tech told me that the restoration would be completed within that hour, but when I got home from work this evening, nearly 10 hours later, my email still wasn’t working, so I called my hosting provider and one of the veteran techs told me that the Level 1 tech had been completely wrong this morning, and the truth is that the complete restoration won’t be completed until sometime late tonight or early tomorrow morning.
I made it clear to that tech that I wouldn’t have minded if someone had told me the truth that it was going to take nearly 24 hours to restore the entire server’s data, but I mind it very, very much that the Level 1 tech told me that the restoration was going to be completed this morning when it actually wasn’t going to be completed until late tonight or early tomorrow morning.
Permalink:
http://www.computerbob.com/wp/what-happened.php
Tags:
Cybercrime, Hardware, Internet, Personal, Security, Tech Support, Web Development