At this point in time, we believe all all services are operating normally. If you find anything awry with your infrastructure, please open an urgent ticket within our support system.
A full RFO detailing this event along with information on service credits will be forthcoming in the next 24 hours.
Thank you
All switches including s43-1 are online and pushing traffic normally. Our staff are combing through customer resources at this time. If you are experiencing an issue with your service, please open an escalated support ticket and we’ll take an immediate look.
Thank you
One of our access layer switches appears to have gone down in the chaos. s43-1.tkwl01 is one of our newer switches; We’re working to get it online again ASAP.
(Web Only Post)
We’re in the process of continuing the clean-up after the outage today.
We’ve just restored services to our highly available load balanced
services (each half of the HA-pair needed to resynchronize with the other
half.) More updates as things progress.
Network traffic appears to be returning to normal. Our initial
investigation seems to indicate that one half of one of our stacked
distribution-layer switches had a failing supervisor module, which around
16:20 PST today created a loop in spanning-tree
We appear to be having a severe problem with our core routing
infrastructure which is creating extremely high packet loss for our entire
network. We are investigating this now and will update this status page as
we learn more.
Our offices will be closed tomorrow, Monday, February 20th, in observance of Presidents’ Day. We will have reduced staff for support issues and limited phone support. As always, our on-call staff will be available for emergencies.
If you have an urgent issue that needs to be addressed while our offices are closed, please make sure to open an urgent ticket or call our phone system and leave a voicemail marking your message as urgent.
To mark your phone call as urgent, please do the following:
To open an urgent support ticket, ensure you are logged into Box Panel and visit http://support.bluebox.net/. Under the ticket body, please check the box “Open this Ticket as Urgent”. This will escalate your ticket and bring it to attention of our team.
Thank you!
(Web Only Post)
We have identified the problem and have a fix in place. All connectivity issues should now be resolved. We are continuing to investigate the root cause, but initial data indicates a failing optic or fiber strand between one distribution router and our core. This event affected a fraction of our data center facilities.
Timeline
As of this time, all services are operating normally. If you are experiencing any issues with the performance or availability or your environment, please utilize our customer chat system and our team will take immediate action.
A full COE will be completed and will be available for distribution in the coming days. Please contact our support team to receive a copy.
A portion of our network is currently experiencing some packet loss. We are investigating and will provide further updates once we have more details.
block-lb02 has been successfully upgraded, and put back into its primary
role. During the transition, load balanced services experienced between 20
and 60 seconds of down-time depending on where they fell in the start-up
order. This concludes the work we will be performing on this server and no
further interruptions are expected. If you are experiencing any further
problems with load balanced services, please escalate a support ticket!