Blue Box System Status

Networking Issue: Updated

Mon, 06 Feb 2012 02:37:52 -0500

From 11:25 to 11:45pm Pacific Time, a small portion of our network experienced packet loss due to a traffic flood generated by a customer’s server. This server has been terminated and all network activity is operating normally at this time. We will continue to monitor the situation closely and our technology and networking teams will be reviewing the event in full detail tomorrow.

Experiencing Networking Issue

Mon, 06 Feb 2012 01:58:00 -0500

This issue is affecting approximately 30% of our network.

Blue Box Support Coverage

Thu, 19 Jan 2012 16:49:38 -0500

With a winter storm affecting travel and power throughout the Northwest, we have limited support coverage at this time. If you have an issue that needs to be addressed immediately, please escalate it to urgent. Our team will respond accordingly.

Our data center is unaffected by the storm and has full Internet connectivity and power.

Load balanced service outage

Fri, 13 Jan 2012 19:28:00 -0500

Today, Friday, January 13th, 2012 at 14:22 PST the primary load balancer in one of our main load balancing clusters crashed during routine maintenance. This crash resulted in the load balancer ceasing to respond to network traffic on all interfaces. Within seconds of this happening the backup load balancer for this cluster came online and began to assume its fail-over role.

This fail-over resulted in a few minutes of downtime for some sites that are balanced by this cluster and as little as a few seconds for others. Our support and technical engineers were alerted of the problem shortly after by our monitoring system and began to diagnose and document the crash. At 14:31 the primary load balancer came back online and our technicians had determined that the machine was stable. At approximately 14:36 the primary load balancer reassumed primary control over the load balancing cluster and traffic was again shifted from the backup load balancer to the primary. This resulted in a brief period of downtime as the services spun up again on the primary. At this time the load balancers are online and stable and as always we will continue to monitor them closely for any problems should they occur.

If you have any questions or notice any anomalies that cause concern please don’t hesitate to contact our Support team.

(Web Only Post)

Virtual Server host vsh156 down

Fri, 13 Jan 2012 14:52:15 -0500

Our monitoring system has informed us that vsh156 is experiencing
problems. This is affecting around 8 Blocks or Virtual Private
Servers on our network. We’re presently engaged in getting the
machine back up, at which point we will be migrating customers off of
it. We expect this issue to be resolved within the next 20-30
minutes.

(Web Only Post)

Connection problems resolved

Sun, 08 Jan 2012 04:59:52 -0500

The link is now back up and running.

Please let us know if you are still seeing any issues.

Connection problems

Sun, 08 Jan 2012 04:27:49 -0500

We’re currently experiencing problems on the link between our primary site
and our nearby backup site. We are in touch with our vendor and we are
currently investigating this issue.

The only services affected by this are backups, which we expect to resume
when the link comes back up.

Please contact support via phone or through Box Panel and escalate a ticket
if you believe you have been affected by this.

All Systems Go

Tue, 03 Jan 2012 23:13:33 -0500

All systems are go at Blue Box.

Blue Box New Years Holiday Coverage

Fri, 30 Dec 2011 18:20:18 -0500

Our offices will be closed Sunday and Monday, 1st and 2nd January, in
observance of the new years holiday. The office will remain closed on
Monday for general business and we will have reduced staff for support
issues and limited phone support. As always, our on-call staff will be
available throughout the holiday and weekend.

If you have an urgent issue that needs to be addressed while our offices
are closed, please make sure to open an urgent ticket or call our phone
system and leave a voicemail marking your message as urgent.

To mark your phone call as urgent, please do the following:

Dial 1-800-613-4305.
At the voice prompt, press 1.
Then press #.
Finally, press 2 and leave a voicemail.

To open an urgent support ticket, ensure you are logged into Box Panel and
visit Under the ticket body, please check the box “Open this Ticket as
Urgent”. This will escalate your ticket and bring
it to attention of our team.

Happy New Year from all of us at the Blue Box Group!

Temporary Office Closure

Thu, 29 Dec 2011 13:52:00 -0500

Note: None of our data center properties were affected by this issue. This only affected our office suite.

The fire alarm in our office building has triggered. Our offices will be temporarily closing until building management has resolved this issue. Inbound phone calls will not presently be answered. In the interim, our offsite teams will be responding to all alerts and escalated tickets.

To mark your phone call as urgent, please do the following:

Dial 1-800-613-4305.
At the voice prompt, press 1.
Then press #.
Finally, press 2 and leave a voicemail.

To open an urgent support ticket, ensure you are logged into Box Panel and visit http://support.bluebox.net/. Under the ticket body, please check the box “Open this Ticket as Urgent”. This will escalate your ticket and bring it to attention of our team.

We will update this page when we are back online in the office.

Update - 11:10AM Pacific - We’ve gotten the all-clear from the Fire Department and our team can now re-enter our office suite. An issue with the HVAC fan belt in our office’s HVAC unit caused the system to billow smoke and odors from vents. The system has been disabled while the HVAC company comes to determine the root cause. There may be a continued lag in our teams general response as there is still a lingering odor, so we’re working on ventilating the space. A portion of our team has relocated to alternative space to support our off site staff. We will update when all staffing is back to normal.

Update - 9:00AM Pacific - As of 2pm yesterday, all staff were back working from the office and the HVAC system was working as expected.

(Web Only Post)

Memcached server is now available again

Tue, 27 Dec 2011 01:01:03 -0500

The issue with the memcached server is now clear and all the sites should
now be working normally again.

Please contact support via phone or through Box Panel and escalate a ticket
if you are still experiencing issues.

Failed memcache server causing limited issues

Tue, 27 Dec 2011 00:38:10 -0500

We’re currently experiencing issues with a memcache server which is causing
login issues to Box Panel and causing issues with some customers sites.

We’re currently getting this corrected, and normal service should be
restored shortly.

Blue Box Holiday Coverage

Fri, 23 Dec 2011 02:22:32 -0500

Blue Box Holiday Coverage:

We’ve heard your feedback loud and clear. We’re excited to announce this
new system status for Blue Box updates.

Our offices will be closed Sunday and Monday, 25th and 26th December, in
observance of the Christmas holiday. The office will remain closed on
Monday for general business and we will have reduced staff for support
issues and limited phone support. As always, our on-call staff will be
available throughout the holiday and weekend.

To mark your phone call as urgent, please do the following:

Dial 1-800-613-4305.
At the voice prompt, press 1.
Then press #.
Finally, press 2 and leave a voicemail.

To open an urgent support ticket, ensure you are logged into Box Panel and
visit Under the ticket body, please check the
box “Open this Ticket as Urgent”. This will escalate your ticket and bring
it to attention of our team.

Merry Christmas from all of us at the Blue Box Group!

All Systems Go

Tue, 13 Dec 2011 19:55:39 -0500

All systems are go at Blue Box!

Significant Box Panel Update on 11/22/11 at 10:30am

Tue, 22 Nov 2011 01:10:00 -0500

Tomorrow morning at 10:30 am Pacific, the Blue Box engineering team will be rolling out a significant update to the Box Panel system. We expect a very brief outage during the update, and those presently logged into Box Panel will need to reconnect. No further issues are expected as part of this update. If you notice anything amiss after the update, please don’t hesitate to open a ticket with our team.

Thanks!

- Blue Box Engineering

System Wide Service Affecting Issue

Fri, 18 Nov 2011 14:55:00 -0500

This morning (11/18/2011), Blue Box experienced a service affecting issue across our installed customer base. There was no break in power to any server, nor any disruption in the physical network. Initial indications point to issues with DNS resolution.

As of 06:45am Pacific time, all services appear to be functioning normally.

The Blue Box engineering team will be doing a full analysis of the issue and will be distributing an RFO to customers around the issue no later than Monday.

Thank you,

- Blue Box Tech Support

100% UP

Thu, 10 Nov 2011 19:58:00 -0500

Like normal!

DNS Resolver Maintenance

Mon, 07 Nov 2011 14:16:00 -0500

We will be switching over to a new primary DNS resolving server to replace our old one at about 1am PDT (09:00 UTC) on Wednesday, November 9th, 2011. There may be a brief DNS resolving disruption lasting for 1-2 seconds. This is usually how long it takes before the resolver client libraries switch to the secondary server.

Please contact support@bluebox.net if you have any questions or concerns about this maintenance.

Boxpanel API Update

Tue, 01 Nov 2011 19:01:00 -0400

Blue Box Group will be testing a major code update to our Boxpanel API starting 10 AM on Wednesday, November 2, 2011. We will be monitoring for errors, but if you see any unusual behavior or activity please do not hesitate to contact us.

stunnel update on Blocks Load Balancers

Tue, 27 Sep 2011 18:44:00 -0400

A potential remote root vulnerability has been found for stunnel versions 4.40 and 4.41 which might possibly affect the earlier version in use on our Blocks Load Balancer product for SSL tunnel termination. As such, we will be rolling out a security update for this software this afternoon. As part of this update we will need to restart the stunnel daemons listening for any load balanced services using SSL offloading (which includes the vast majority of load balanced services using the https:// protocol). No functionality changes are expected as a result of this upgrade. However, restarting the service means that each SSL-offloaded load balanced service will experience a < 1 second “hiccup” as the stunnel service is restarted. Please contact support@bluebox.net if you have questions about this.