At approximately 8:30am EST this morning, all Articulate Online hosted accounts went offline. All accounts were back online as of about 4:30pm EST. The problem was related to a network/IP address configuration issue at our data center. The good news:
No data was lost and no accounts were compromised.
We’re Very Sorry
We’re really sorry for this downtime. We know you and your customers count on us for your elearning programs, and we hate to let you down like we did today. We accept full responsibility for what happened. We’re investigating the technical specifics of what went wrong and are really trying to understand what caused this problem. We’ll also be evaluating our disaster preparedness plan to avoid something like this in the future.
What Went Wrong
Here’s the rough timeline of events that we experienced today:
- At 8:30am EST this morning, our server monitoring application alerted us that there was a problem with the primary Articulate Online server, where all hosted accounts reside.
- Our dev team started researching the issue shortly thereafter.
- Around 11:30am EST, our data center technicians identified the cause of the issue (network/IP-related) and began working toward resolution.
- At 1:30pm EST, we were able to reroute some of the incoming AO traffic to a service announcement page hosted on a different server.
- By 4:30pm, we had restored the AO DNS entries to their correct values and all accounts and services were functioning as expected.
Thank You for Your Patience
We know how frustrating it is when a product you depend on to run your business is suddenly not there. So thank you for hanging in there with us. This was an important experience for us that we’re taking very seriously. We’re committed to improving our server infrastructure to minimize the chances of this happening again.
In the past year since
AO went live, we’ve experienced minimal downtime, most of which was a result of brief, planned downtimes for product upgrades.
As always, please feel free to
contact me directly with any specific questions or concerns (or discuss your concerns in this forum thread).