Blue Box System Status

All Services Operating Normally

Mon, 20 Feb 2012 22:04:58 -0500

At this point in time, we believe all all services are operating normally. If you find anything awry with your infrastructure, please open an urgent ticket within our support system.

A full RFO detailing this event along with information on service credits will be forthcoming in the next 24 hours.

Thank you

Network Outage Update

Mon, 20 Feb 2012 21:00:11 -0500

All switches including s43-1 are online and pushing traffic normally. Our staff are combing through customer resources at this time. If you are experiencing an issue with your service, please open an escalated support ticket and we’ll take an immediate look.

Thank you

Network outage update

Mon, 20 Feb 2012 20:41:00 -0500

One of our access layer switches appears to have gone down in the chaos. s43-1.tkwl01 is one of our newer switches; We’re working to get it online again ASAP.

(Web Only Post)

Clean-up continues...

Mon, 20 Feb 2012 20:26:27 -0500

We’re in the process of continuing the clean-up after the outage today.
We’ve just restored services to our highly available load balanced
services (each half of the HA-pair needed to resynchronize with the other
half.) More updates as things progress.

Update on outage

Mon, 20 Feb 2012 20:00:19 -0500

Network traffic appears to be returning to normal. Our initial
investigation seems to indicate that one half of one of our stacked
distribution-layer switches had a failing supervisor module, which around
16:20 PST today created a loop in spanning-tree

Severe network issues

Mon, 20 Feb 2012 19:41:25 -0500

We appear to be having a severe problem with our core routing
infrastructure which is creating extremely high packet loss for our entire
network. We are investigating this now and will update this status page as
we learn more.

Presidents' Day Office Closure

Mon, 20 Feb 2012 01:21:00 -0500

Our offices will be closed tomorrow, Monday, February 20th, in observance of Presidents’ Day. We will have reduced staff for support issues and limited phone support. As always, our on-call staff will be available for emergencies.

If you have an urgent issue that needs to be addressed while our offices are closed, please make sure to open an urgent ticket or call our phone system and leave a voicemail marking your message as urgent.

To mark your phone call as urgent, please do the following:

Dial 1-800-613-4305.
At the voice prompt, press 1.
Then press #.
Finally, press 2 and leave a voicemail.

To open an urgent support ticket, ensure you are logged into Box Panel and visit http://support.bluebox.net/. Under the ticket body, please check the box “Open this Ticket as Urgent”. This will escalate your ticket and bring it to attention of our team.

Thank you!

(Web Only Post)

Network connectivity issues [RESOLVED]

Thu, 16 Feb 2012 06:33:00 -0500

We have identified the problem and have a fix in place. All connectivity issues should now be resolved. We are continuing to investigate the root cause, but initial data indicates a failing optic or fiber strand between one distribution router and our core. This event affected a fraction of our data center facilities.

Timeline

2:45am - Initial flap of one redundant link between distribution layer and core reported in our logs.
2:50am - Initial Blue Box monitoring alerts our staff. First customer reports of connectivity issues arrive.
3:04am - Full Sr. Technology and Networking teams from Blue Box online and diagnosing the issue. Initial investigation indicates an isolated event to a single distribution switch and its connection back to the core.
3:22am - The flapping link is brought offline, sending all traffic through the redundant link. All traffic appears to be flowing normally at this time and all connectivity and packet loss issues clear up.

As of this time, all services are operating normally. If you are experiencing any issues with the performance or availability or your environment, please utilize our customer chat system and our team will take immediate action.

A full COE will be completed and will be available for distribution in the coming days. Please contact our support team to receive a copy.

Network connectivity issues

Thu, 16 Feb 2012 06:17:17 -0500

A portion of our network is currently experiencing some packet loss. We are investigating and will provide further updates once we have more details.

Load-balancer upgrade complete

Wed, 15 Feb 2012 02:31:29 -0500

block-lb02 has been successfully upgraded, and put back into its primary
role. During the transition, load balanced services experienced between 20
and 60 seconds of down-time depending on where they fell in the start-up
order. This concludes the work we will be performing on this server and no
further interruptions are expected. If you are experiencing any further
problems with load balanced services, please escalate a support ticket!

Load balancer update.

Tue, 14 Feb 2012 19:55:08 -0500

We’ve been able to determine that block-lb02 experienced a partial failure
wherein those portions of its software stack which load balance requests
stopped functioning due to a memory starvation issue, but those portions of
the stack responding to heartbeat requests didn’t stop working; meaning
that the other unit in this highly-available pair (block-lb01) did not take
over as it was unable to detect this partial failure. This continued for
about 5 minutes until we pulled block-lb02 off the network, which caused
block-lb01 to detect the failure and take over.

We will be upgrading block-lb02 today and scheduling a brief window tonight
at 23:00 PDT (07:00 UTC) to put it back in the loop and fail services back
over to it so we can upgrade block-lb01. During this secondary failover,
some services may experience up to 20 seconds of down-time as the load
balanced services move to the other host.

All of our other load balancers are built with more RAM than these two, so
should not be in danger of any memory-starvation related failures in the
immediate future. In addition to this, over the next week we will be
putting better detection in place to catch low RAM scenarios on the load
balancers before they are likely to cause an outage.

(Web only post)

Load Balancer outage update.

Tue, 14 Feb 2012 18:47:51 -0500

The load balancer outage referenced in the latest update also affected a
number of customer applications. It appears that the load balancer was
timing out on processing requests, however since it didn’t go down
completely and was still available for its health check, it didn’t fail
over to its redundant pair. Now that things are back online, we are
investigating the cause of this issue, as well as ways to ensure that
should similar issues arise in the future, failover will happen in a more
timely fashion.

We’ll provide an update once we have more information.

Boxpanel unavailable for 10 minutes

Tue, 14 Feb 2012 18:29:59 -0500

Our Box Panel application was offline for 10 minutes today (3:12pm - 3:22pm
PT), during which time API was also likely offline.

Our load balancer was in a non-responsive state, but after a reboot all
appears normal.

Emergency Networking Upgrade Complete

Sat, 11 Feb 2012 07:57:06 -0500

We have completed the emergency network upgrade.

If you see any abnormal latency, reduction in service or dropped packets please open an escalated ticket with our support team or give us a call at 800.613.4305.

Emergency Networking Update: 0200 PST Status

Sat, 11 Feb 2012 05:24:47 -0500

We’re about 65-70% complete with the networking work. There have been some
BGP issues with our upstream link that has caused some short outages for
sites, and some internal traffic between servers has been slower than usual.

We’ll be getting a complete wrap up out early next week.

Emergency Network upgrade - updated

Sat, 11 Feb 2012 04:01:39 -0500

The network upgrade is proceeding according to plan. There is the the
potential for diminished functionality to some hosts when some changes take
effect.

If you are noticing connectivity issues, please let us know at
support@bluebox.net or via customer chat.

Network Issue this morning.

Fri, 10 Feb 2012 14:07:31 -0500

From 10:35 - 10:55 a portion of our network experienced some packet loss
due to a distributed denial of service attack (DDoS) directed at a server
on our network. We have addressed the issue, and any network connectivity
issues should now be resolved. If you continue to experience any network
problems, please get in touch with our support team, and we’ll address it
as best we can.

Emergency Network Upgrade

Thu, 09 Feb 2012 18:43:24 -0500

We will be performing an emergency upgrade of our core routers and the distribution layer of our network from GigE to 10-GigE on Friday, February 10th, 2012 starting at 23:00 PST. This network upgrade will address network performance issues that have been raised and alleviate network congestion. No downtime should be incurred as a result of this upgrade. We will have our senior administrators as well as our entire NOC and network engineering team on site during this work to both assist in the upgrade and to address any complications, should they arise. This upgrade is scheduled to take 5 hours, and again no downtime or significant degradation of network service is expected.

If you have any questions, concerns or if there is anything that we can address, please open a ticket with our support team or give us a call at 800.613.4305.

All Systems Go

Thu, 09 Feb 2012 12:50:29 -0500

All systems are go at Blue Box.

Networking Issue: Updated

Mon, 06 Feb 2012 02:37:52 -0500

From 11:25 to 11:45pm Pacific Time, a small portion of our network experienced packet loss due to a traffic flood generated by a customer’s server. This server has been terminated and all network activity is operating normally at this time. We will continue to monitor the situation closely and our technology and networking teams will be reviewing the event in full detail tomorrow.