Today, Friday, January 13th, 2012 at 14:22 PST the primary load balancer in one of our main load balancing clusters crashed during routine maintenance. This crash resulted in the load balancer ceasing to respond to network traffic on all interfaces. Within seconds of this happening the backup load balancer for this cluster came online and began to assume its fail-over role.

This fail-over resulted in a few minutes of downtime for some sites that are balanced by this cluster and as little as a few seconds for others. Our support and technical engineers were alerted of the problem shortly after by our monitoring system and began to diagnose and document the crash. At 14:31 the primary load balancer came back online and our technicians had determined that the machine was stable. At approximately 14:36 the primary load balancer reassumed primary control over the load balancing cluster and traffic was again shifted from the backup load balancer to the primary. This resulted in a brief period of downtime as the services spun up again on the primary. At this time the load balancers are online and stable and as always we will continue to monitor them closely for any problems should they occur.

If you have any questions or notice any anomalies that cause concern please don’t hesitate to contact our Support team.

(Web Only Post)