We have identified the problem and have a fix in place. All connectivity issues should now be resolved. We are continuing to investigate the root cause, but initial data indicates a failing optic or fiber strand between one distribution router and our core.  This event affected a fraction of our data center facilities.

Timeline

• 2:45am - Initial flap of one redundant link between distribution layer and core reported in our logs.  
• 2:50am - Initial Blue Box monitoring alerts our staff.  First customer reports of connectivity issues arrive.
• 3:04am - Full Sr. Technology and Networking teams from Blue Box online and diagnosing the issue.  Initial investigation indicates an isolated event to a single distribution switch and its connection back to the core.
• 3:22am - The flapping link is brought offline, sending all traffic through the redundant link.  All traffic appears to be flowing normally at this time and all connectivity and packet loss issues clear up.

As of this time, all services are operating normally.  If you are experiencing any issues with the performance or availability or your environment, please utilize our customer chat system and our team will take immediate action.

A  full COE will be completed and will be available for distribution in the coming days. Please contact our support team to receive a copy.