I just received some of the most impressive customer service I have ever experienced. Last night, one of our Rackspace Cloud Servers was down for three periods totaling several hours. Every time the server went down, I promptly received an email from Rackspace which clearly acknowledged the problem, explained the cause, told me what action Rackspace was taking, and provided an estimated time to repair the problem. Every time the server came back up, I received an email telling me so

This morning, I received a message explaining in detail why the server was down for such an extended period and recapping the problem history and resolution. It also included this:

We want to apologize for any inconvenience this may have caused. Our team will proactively provide you with compensation for your downtime per our SLA

Finally, I received a voice mail message from Rackspace this afternoon. An account manager called me, clearly explained the issue, apologized for the downtime, and reiterated that they had proactively credited our account for the full month’s hosting fees.

This is truly remarkable for three reasons. First, Rackspace provided all of the information I could possibly have wanted and they did so more quickly than I could call them and request it. Second, Rackspace honored the SLA without requiring any action on my part whatsoever. Third, in the aftermath, Rackspace reached out to me not just in writing but with a personal phone call.

Our new HTN Cloud Hosting accounts are built on top of these Rackspace Cloud Servers. This experience reinforces my judgement that choosing Rackspace as our partner was an excellent decision. I am quite confident that the HTN Cloud Hosting family will provide managed hosting which is truly second to none.

We are occasionally asked why Hen’s Teeth Network does not offer a Service Level Agreement or “SLA” on our hosting products. We do not offer an SLA for two reasons:

1. If your web site is down, the cost of hosting and the monetary credit you receive under most SLA’s is an insignificant portion of the lost revenue.
2. Until you move into the marketplace for true high availability computing, the SLA’s offered by most hosting companies have so many exceptions that they are worthless.

In lieu of an SLA, we offer a 100% satisfaction guarantee.

Hen’s Teeth Network is a gold level reseller of Verio’s web hosting products. We have been hosting exclusively with Verio since 2002, have been extremely pleased with the service, and recommend nothing else for our clients. I think that Verio’s SLA is typical of the industry so I will use it as an example. Were we to offer an SLA, it would follow this one exactly.

First, the numbers: There are 730.5 hours in an average month so the maximum downtime is calculated from that figure.

• 99.9 to 100% availability – maximum of 0:43 downtime – 0% credit
• 98% to 99.8% – 14:36 downtime – 10% credit
• 95% to 97.9% – 36:31 (1.5 days) downtime – 20% credit
• 90% to 94.9% – 73:03 (3 days) downtime – 50% credit
• 89.9% or below – 100% credit

If the machine crashes and Verio reboots it and runs fsck to clean up the file system and it is back in 40 minutes (a very reasonable expectation), you get nothing. If it takes longer to reboot, up to 14 hours, which is more than a full business day, you get a whopping 10% credit.

Let’s take that a step further and look at two typical payment scenarios. Many of our clients use our Signature hosting accounts which cost $19 per month. Would anyone be served if we credited their account, or sent them a check, for $1.90? Some of our clients use Managed Private Servers (dedicated servers) which cost $325 per month. Were we to send them $32.50, would it even begin to make up for the lost business of being down for a business day?

Second, the exceptions: Verio’s SLA excludes (and this is typical of the industry), “scheduled maintenance and emergency maintenance and upgrades” and “circumstances beyond Verio’s reasonable control including… attacks or hackers….” In other words, if the downtime is caused by “maintenance,” even if the maintenance takes much longer than expected and runs from the middle-of-the-night-maintenance-window into the following business day, it doesn’t count. More to the point, if the downtime is caused by hackers, it doesn’t count. This hacker exclusion means that if your web site is infested with malware and becomes unusable; or if some miscreant launches a DOS (denial-of-service) attack against another site hosted on the same physical machine as hosts your own site, and your site becomes unusable, it doesn’t count.

In the 15 years that I have been hosting web sites in various capacities, downtime is virtually never caused by the sorts of things covered by SLA’s. They offer a false sense of security.

In the seven years that I have been hosting with Verio, the reliability has been outstanding. Irrespective of the SLA and the exclusions, our clients’ web sites and email have simply “been there” when needed. I have every expectation that that will continue to be true. Exceptional reliability is one of the key reasons why I selected Verio as our hosting provider.

Imagine this: You wake up to find that your web server and your backup server have both been hacked. You have lost all of the data that your company has spent the last 13 years accumulating. Because of poor choices for backups, that just happened to AVSIM.

I have been doing system administration for over 30 years and I can be fanatical about the backups. (Ask any of the folks who work for me how testy I get when their machines are not fully backed up daily.) I have been burned by faulty backups and know the pain. Back in 2001, my laptop was stolen and everything was backed up except my Zemon family genealogy research. It took years to recover from that disaster. My first hard lesson came in 1980 when our PDP-11/70 crashed and I had to fly from Los Angeles to San Jose so that I could use a sister project’s similarly equipped PDP-11/70 to rebuild our boot disk. That was an expensive crash: a dozen people lost use of their computer for three business days.

I chose Verio as our hosting partner for many reasons, one of which is the excellent backups provided with every hosting account.

1. All of the disk drives are “mirrored,” meaning that if one drive fails, the computer keeps running using the second drive; no data are lost.
2. Every night, the entire machine is copied to a backup disk and the backup remains available as “read-only” files until the next night when they are overwritten by a new backup. Anyone hosting a web site from us can recover files from these nightly backups for free. I cannot count the number of times these backups have saved the day when the phone rang and the panicked voice on the other end said, “I messed up and deleted a file….”
3. Every week, the entire machine is copied to tape and two or three weeks’ worth of tapes are kept in a vault. These, too, are accessible by clients though there is a fee to cover the labor of manually retrieving the tape from the vault. These, too, have saved several clients’ butts over the years.

For clients who want more security, we offer an add-on service, Nest Egg Backup for Web Servers, which provides 30 days of backups of every file on the server.

Few other hosting companies can help you if you accidentally delete a file or need to quickly put your web site back to the way it was yesterday. If you don’t know whether your hosting provider protects you, call and ask them.

Update: Scott and Candy just pointed out a problem with the home page of our site which had been there for several weeks without anyone here noticing it. (Who looks at the home page of his own web site?) Using Nest Egg Backup for Web Servers, I quickly restored the missing code from a file which has not been on our server since April 23, almost a month ago.

How important are backups to you? How would you feel if you lost your entire web site could not get it back?

It is not often that I have the pleasure of introducing a completely new family of hosting accounts but today is one of those special days. See Verio CP/M for full details. While priced for today’s economy, these new plans remain surprisingly feature-rich.