We have gotten a full report from our network provider as to the event that happened this morning between 04:10 and 04:20 PDT. They have confirmed that there was a full disruption to network connectivity between 04:12 and 04:13 PDT and that this will have affected traffic for a few moments around that timeframe. The cause of this outage was due to a faulty optical cable which is going to be replaced tonight. The outage was exacerbated because the failure entailed an unexpected BGP re-convergence event, delayed by the failure mode that happened (ie. “link” wasn’t dropped, so the BGP session had to timeout before our routers recognized the upstream as being down.): At this point our network traffic started routing over alternate carriers, but unfortunately it does take a little bit for this to start happening.

The replacement of the faulty optical cable will not affect service as they will be routing around that section of optical cable prior to replacing it. Furthermore they will be installing a redundant 10G circuit that bridges the same routers within the next 10 days in order to prevent outages of this nature from occurring in the future.

If you have any question or concerns please contact our support team via phone, chat or ticket and we will gladly address them for you.