Articulate Online Back Online - Downtime Explained

[jtarnoff]

Thank you David and Gabe; I appreciate you taking the time to respond to my questions.

While I don't mean to be difficult, I don't really "like" these answers. I'm still waiting for an email announcing the downtime and what was done about it. This needs to be sent NOW to ALL of your customers. RSS and a status page are nice--but frankly not even necessary. The problem with those avenues of communication is that they require your users to KNOW something is wrong before checking (or to take the time to subscribe to an RSS feed). There should be a PROACTIVE communication from Articulate.

Also, from some quick checking, it appears that you use ZoneEdit for your DNS services and aplus.net for your server hosting. Is that correct? Having been a customer of both, neither are geared toward mission-critical applications. Both are primarily budget providers (neither have very good SLAs). I would expect you to be spending the extra money and using services like Xiolink, Rackspace, or DataPipe for hosting and UltraDNS for DNS.

What EXACTLY is being done to prevent these issues from reoccuring? Will you be setting up mirrored servers in a different datacenter? Will you be migrating to a more reliable data center?

I am a small business owner, and as I said earlier, I completely understand that mistakes happen. But, at the end of the day, what really matters is how the mistake is dealt with and what is done to prevent it in the future. My staff spent a great deal of time yesterday fielding phone calls from unhappy employees of our client, procuring a new online testing service, inputting and launching a 100 item test, and then communicating to all of these employees to let them know there was an alternate testing site available. While this all costs time and money, my biggest issue is that when people called, all I could say, is "I'm sorry, I know it's down; I don't know why, and I don't know when it will be fixed." Needless to say, this does not inspire confidence in the minds of our clients who pay us millions of dollars annually for our training programs. It doesn't matter that the actual test is the smallest part of the program. As you undoubtedly know, when something goes wrong, IT is all of a sudden the most important component.

[gabe]

Thanks for sharing your additional concerns and questions, jtarnoff. We really appreciate your feedback, and our entire management team is aware of your concerns and input.

The reason we didn’t email everyone when this happened is that, as it turns out, only about 12% of customers were impacted by the downtime. While we knew only a small percentage were being impacted, we didn’t know at the time exactly which accounts were being impacted. We didn’t want to unnecessarily alarm the entire population, especially when we were under the impression that the issue would be resolved in minutes.

As for our data center, we’re in the process of migrating all accounts to Rackspace so that this doesn't happen again. We actually already host other servers with Rackspace, including the ones that we’re typing on right now, so we have an existing relationship with them and know how great they are. We expect this migration to be fully complete in the next 30 days and will update you when it’s done.

[gabe]

By the way, you should have also received the status email I just sent out with an explanation of yesterday's events.

[Spectre]

I just received the e-mail from Gabe, and I'm pleased to see that the servers are getting migrated to a more reliable data center.

Because of this outage and the fact that our company has used AO for a brief period, I am concerned about my management's perception of AO's reliability. I anticipate receiving inquiries from management and our IT folks regarding the reliability of AO and specifically the SLA.

Is this information something you can disseminate when the migration is complete? We like AO very much, but we need to give management some reassurance that there is nothing to worry about.

Thanks,
Phil

[gabe]

Thanks, Phil. We definitely understand your concerns about the perception issue.

Yes, we will keep you posted with regard to the Rackspace migration and other efforts around this.

[gabe]

Good news: We've launched the AO status monitoring site!

[awkeogh]

Any explanation for today's significant outage?

[gabe]

Hi There-

You can see a record of events as they unfolded at heartbeat.articulate.com. Here's the specific post that outlines what happened:

2:26 AM PST on Jan 17, 2008: According to our data center one of the routers in the data center has failed and they are currently in the process of replacing the failed router. We will continue to keep you posted as we receive updates.

[JK_Certegy]

In all fairness, (a concept that might be absent from your business philosophy, and understandably so) the manner in which Articulate chose to communicate the specifics about this issue, is really rather exceptional when compared to the manner in which many service providers in general, and their contemporaries in specific, tend to deal with failings of this nature. Compare the personally drafted letter we received to the next communication your ISP offers when your connectivity disappears. Compare the hold time you’ll spend waiting for their call center to tell you they are aware of the issue, with the nearly immediate email you would have received from the folks in Articulate’s customer support had you checked the support page. I had barely finished forwarding the notification, when I received a note from a senior support rep. indicating the issue was under investigation. You’d get no such thing from a BlackBoard or some other such hosted service.

Given the that Articulate is not obligated to be so transparent regarding the specifics of the event, the fact that they’ve admitted the error, and detailed their preventative measures going forward, is for my constituents, more than satisfactory. I work for a HUGE corporation with many shared and contracted services that aren’t supported nearly as well as Articulate Online is. Their service is the ONLY reason we don’t host all of our training content (in the thousands of hours) internally, as our own IT elements would have it. They simply can’t provide the level of service that Gabe and his staff are able to.

I’m not an Articulate mark. I have plenty of criticism for the products, but it the service is top notch in my opinion. Systems are not infallible, and that’s why you gather talented and creative people around you to ensure these things are kept to a minimum – as they have. Do you personally write letters or call each of your customers when your systems fail - even for what was anticipated as a brief outage? If you paid that much attention to your offering, perhaps you’d have noticed the system was down –before you got complaints. Being a small business owner focused on the millions your customers pay you, I imagine it's easy to sit back and wave the "customer's always right" flag and demand absolute transparency into thier strategy. In the real world, fixing the problem and being applogetic about it are usually mutually exclusive. Get used to it - it's the cost of doing business.

Originally Posted by jtarnoff

Thank you David and Gabe; I appreciate you taking the time to respond to my questions.

While I don't mean to be difficult, I don't really "like" these answers. I'm still waiting for an email announcing the downtime and what was done about it. This needs to be sent NOW to ALL of your customers. RSS and a status page are nice--but frankly not even necessary. The problem with those avenues of communication is that they require your users to KNOW something is wrong before checking (or to take the time to subscribe to an RSS feed). There should be a PROACTIVE communication from Articulate.

Also, from some quick checking, it appears that you use ZoneEdit for your DNS services and aplus.net for your server hosting. Is that correct? Having been a customer of both, neither are geared toward mission-critical applications. Both are primarily budget providers (neither have very good SLAs). I would expect you to be spending the extra money and using services like Xiolink, Rackspace, or DataPipe for hosting and UltraDNS for DNS.

What EXACTLY is being done to prevent these issues from reoccuring? Will you be setting up mirrored servers in a different datacenter? Will you be migrating to a more reliable data center?

I am a small business owner, and as I said earlier, I completely understand that mistakes happen. But, at the end of the day, what really matters is how the mistake is dealt with and what is done to prevent it in the future. My staff spent a great deal of time yesterday fielding phone calls from unhappy employees of our client, procuring a new online testing service, inputting and launching a 100 item test, and then communicating to all of these employees to let them know there was an alternate testing site available. While this all costs time and money, my biggest issue is that when people called, all I could say, is "I'm sorry, I know it's down; I don't know why, and I don't know when it will be fixed." Needless to say, this does not inspire confidence in the minds of our clients who pay us millions of dollars annually for our training programs. It doesn't matter that the actual test is the smallest part of the program. As you undoubtedly know, when something goes wrong, IT is all of a sudden the most important component.