February 2012
21 posts
All Services Operating Normally
At this point in time, we believe all all services are operating normally. If you find anything awry with your infrastructure, please open an urgent ticket within our support system.
A full RFO detailing this event along with information on service credits will be forthcoming in the next 24 hours.
Thank you
Network Outage Update
All switches including s43-1 are online and pushing traffic normally. Our staff are combing through customer resources at this time. If you are experiencing an issue with your service, please open an escalated support ticket and we’ll take an immediate look.
Thank you
Network outage update
One of our access layer switches appears to have gone down in the chaos. s43-1.tkwl01 is one of our newer switches; We’re working to get it online again ASAP.
(Web Only Post)
Clean-up continues...
We’re in the process of continuing the clean-up after the outage today.
We’ve just restored services to our highly available load balanced
services (each half of the HA-pair needed to resynchronize with the other
half.) More updates as things progress.
Update on outage
Network traffic appears to be returning to normal. Our initial
investigation seems to indicate that one half of one of our stacked
distribution-layer switches had a failing supervisor module, which around
16:20 PST today created a loop in spanning-tree
Severe network issues
We appear to be having a severe problem with our core routing
infrastructure which is creating extremely high packet loss for our entire
network. We are investigating this now and will update this status page as
we learn more.
Presidents' Day Office Closure
Our offices will be closed tomorrow, Monday, February 20th, in observance of Presidents’ Day. We will have reduced staff for support issues and limited phone support. As always, our on-call staff will be available for emergencies.
If you have an urgent issue that needs to be addressed while our offices are closed, please make sure to open an urgent ticket or call our phone system and leave...
Network connectivity issues [RESOLVED]
We have identified the problem and have a fix in place. All connectivity issues should now be resolved. We are continuing to investigate the root cause, but initial data indicates a failing optic or fiber strand between one distribution router and our core. This event affected a fraction of our data center facilities.
Timeline
2:45am - Initial flap of one redundant link between distribution...
Network connectivity issues
A portion of our network is currently experiencing some packet loss. We are investigating and will provide further updates once we have more details.
Load-balancer upgrade complete
block-lb02 has been successfully upgraded, and put back into its primary
role. During the transition, load balanced services experienced between 20
and 60 seconds of down-time depending on where they fell in the start-up
order. This concludes the work we will be performing on this server and no
further interruptions are expected. If you are experiencing any further
problems with load...
Load balancer update.
We’ve been able to determine that block-lb02 experienced a partial failure
wherein those portions of its software stack which load balance requests
stopped functioning due to a memory starvation issue, but those portions of
the stack responding to heartbeat requests didn’t stop working; meaning
that the other unit in this highly-available pair (block-lb01) did not take
over as it...
Load Balancer outage update.
The load balancer outage referenced in the latest update also affected a
number of customer applications. It appears that the load balancer was
timing out on processing requests, however since it didn’t go down
completely and was still available for its health check, it didn’t fail
over to its redundant pair. Now that things are back online, we are
investigating the cause of this...
Boxpanel unavailable for 10 minutes
Our Box Panel application was offline for 10 minutes today (3:12pm - 3:22pm
PT), during which time API was also likely offline.
Our load balancer was in a non-responsive state, but after a reboot all
appears normal.
Emergency Networking Upgrade Complete
We have completed the emergency network upgrade.
If you see any abnormal latency, reduction in service or dropped packets please open an escalated ticket with our support team or give us a call at 800.613.4305.
Emergency Networking Update: 0200 PST Status
We’re about 65-70% complete with the networking work. There have been some
BGP issues with our upstream link that has caused some short outages for
sites, and some internal traffic between servers has been slower than usual.
We’ll be getting a complete wrap up out early next week.
Emergency Network upgrade - updated
The network upgrade is proceeding according to plan. There is the the
potential for diminished functionality to some hosts when some changes take
effect.
If you are noticing connectivity issues, please let us know at
support@bluebox.net or via customer chat.
Network Issue this morning.
From 10:35 - 10:55 a portion of our network experienced some packet loss
due to a distributed denial of service attack (DDoS) directed at a server
on our network. We have addressed the issue, and any network connectivity
issues should now be resolved. If you continue to experience any network
problems, please get in touch with our support team, and we’ll address it
as best we can.
Emergency Network Upgrade
We will be performing an emergency upgrade of our core routers and the distribution layer of our network from GigE to 10-GigE on Friday, February 10th, 2012 starting at 23:00 PST. This network upgrade will address network performance issues that have been raised and alleviate network congestion. No downtime should be incurred as a result of this upgrade. We will have our senior administrators as...
All Systems Go
All systems are go at Blue Box.
Networking Issue: Updated
From 11:25 to 11:45pm Pacific Time, a small portion of our network experienced packet loss due to a traffic flood generated by a customer’s server. This server has been terminated and all network activity is operating normally at this time. We will continue to monitor the situation closely and our technology and networking teams will be reviewing the event in full detail tomorrow.
Experiencing Networking Issue
This issue is affecting approximately 30% of our network.
January 2012
6 posts
Blue Box Support Coverage
With a winter storm affecting travel and power throughout the Northwest, we have limited support coverage at this time. If you have an issue that needs to be addressed immediately, please escalate it to urgent. Our team will respond accordingly.
Our data center is unaffected by the storm and has full Internet connectivity and power.
Load balanced service outage
Today, Friday, January 13th, 2012 at 14:22 PST the primary load balancer in one of our main load balancing clusters crashed during routine maintenance. This crash resulted in the load balancer ceasing to respond to network traffic on all interfaces. Within seconds of this happening the backup load balancer for this cluster came online and began to assume its fail-over role.
This fail-over...
Virtual Server host vsh156 down
Our monitoring system has informed us that vsh156 is experiencing
problems. This is affecting around 8 Blocks or Virtual Private
Servers on our network. We’re presently engaged in getting the
machine back up, at which point we will be migrating customers off of
it. We expect this issue to be resolved within the next 20-30
minutes.
(Web Only Post)
Connection problems resolved
The link is now back up and running.
Please let us know if you are still seeing any issues.
Connection problems
We’re currently experiencing problems on the link between our primary site
and our nearby backup site. We are in touch with our vendor and we are
currently investigating this issue.
The only services affected by this are backups, which we expect to resume
when the link comes back up.
Please contact support via phone or through Box Panel and escalate a ticket
if you believe you have...
All Systems Go
All systems are go at Blue Box.
December 2011
6 posts
Blue Box New Years Holiday Coverage
Our offices will be closed Sunday and Monday, 1st and 2nd January, in
observance of the new years holiday. The office will remain closed on
Monday for general business and we will have reduced staff for support
issues and limited phone support. As always, our on-call staff will be
available throughout the holiday and weekend.
If you have an urgent issue that needs to be addressed while our...
Temporary Office Closure
Note: None of our data center properties were affected by this issue. This only affected our office suite.
The fire alarm in our office building has triggered. Our offices will be temporarily closing until building management has resolved this issue. Inbound phone calls will not presently be answered. In the interim, our offsite teams will be responding to all alerts and escalated tickets.
...
Memcached server is now available again
The issue with the memcached server is now clear and all the sites should
now be working normally again.
Please contact support via phone or through Box Panel and escalate a ticket
if you are still experiencing issues.
Failed memcache server causing limited issues
We’re currently experiencing issues with a memcache server which is causing
login issues to Box Panel and causing issues with some customers sites.
We’re currently getting this corrected, and normal service should be
restored shortly.
Blue Box Holiday Coverage →
We’ve heard your feedback loud and clear. We’re excited to announce this
new system status for Blue Box updates.
Our offices will be closed Sunday and Monday, 25th and 26th December, in
observance of the Christmas holiday. The office will remain closed on
Monday for general business and we will have reduced staff for support
issues and limited phone support. As always, our...
All Systems Go
All systems are go at Blue Box!
November 2011
5 posts
Significant Box Panel Update on 11/22/11 at...
Tomorrow morning at 10:30 am Pacific, the Blue Box engineering team will be rolling out a significant update to the Box Panel system. We expect a very brief outage during the update, and those presently logged into Box Panel will need to reconnect. No further issues are expected as part of this update. If you notice anything amiss after the update, please don’t hesitate to open a ticket...
System Wide Service Affecting Issue
This morning (11/18/2011), Blue Box experienced a service affecting issue across our installed customer base. There was no break in power to any server, nor any disruption in the physical network. Initial indications point to issues with DNS resolution.
As of 06:45am Pacific time, all services appear to be functioning normally.
The Blue Box engineering team will be doing a full analysis of...
100% UP
Like normal!
DNS Resolver Maintenance
We will be switching over to a new primary DNS resolving server to replace our old one at about 1am PDT (09:00 UTC) on Wednesday, November 9th, 2011. There may be a brief DNS resolving disruption lasting for 1-2 seconds. This is usually how long it takes before the resolver client libraries switch to the secondary server.
Please contact support@bluebox.net if you have any questions or concerns...
Boxpanel API Update
Blue Box Group will be testing a major code update to our Boxpanel API starting 10 AM on Wednesday, November 2, 2011. We will be monitoring for errors, but if you see any unusual behavior or activity please do not hesitate to contact us.
September 2011
3 posts
stunnel update on Blocks Load Balancers
A potential remote root vulnerability has been found for stunnel versions 4.40 and 4.41 which might possibly affect the earlier version in use on our Blocks Load Balancer product for SSL tunnel termination. As such, we will be rolling out a security update for this software this afternoon. As part of this update we will need to restart the stunnel daemons listening for any load balanced services...
Office closed for company meeting - 9/20
Our offices will be closed from 3:30pm Pacific - 6pm Pacific today for a company meeting. During this time, if you have an urgent issue, please escalate your ticket and our team will respond. Otherwise, we will be available for general support starting at 7pm Pacific.
Thank you,
- Blue Box Group
Core Database Maintenance
Tonight starting at 11:00pm PDT (06:00 UTC), we will be performing maintenance on the core database infrastructure which powers Box Panel and our Support system. For the approximately 10-15 minutes it will take to do this work, Box Panel, the Blue Box Group API interface, and our support system will be offline. We do not expect this work to affect any customer machines.
August 2011
3 posts
Upstream Transit Issue
On Monday, August 8th at ~16:07 PDT (UTC -06:00) one of our upstream providers had a brief outage while one of their routers unexpectedly reset. During a brief ~5 minute time frame some users may have experienced additional latency or possible dropped packets. The upstream provider will be implementing a permanent fix in the near future for this problem, until then we don’t expect any...
Upstream Network Provider Maintenance 08.20.11
On Saturday evening, August 20th until Sunday morning, August 21st 2011 One of our Upstream Network Providers
will be performing scheduled maintenance on one of our Westcoast transit pipes.
Maintenance will begin at 11:55 PM PDT Saturday evening and end by 4:00 AM
PDT Sunday morning. There is no downtime anticipated, but with any network maintenance there is a heightened risk of downtime.
Network Uplink Maintenance
On Monday August 8th, 2011 at 11:00 PM PDT (06:00 UTC) and lasting for up to three hours, we will be adding some additional networking gear, we expect this maintenance to have no significant impact on our customers or their running applications. However, as is the nature of any maintenance on the physical connections to our core routing infrastructure, an increased risk of an outage exists.
July 2011
2 posts
Office closed for company meeting - 7/14
Our offices will be closed from 4:30pm Pacific - 6pm Pacific today for a company meeting. During this time, if you have an urgent issue, please escalate your ticket and our team will respond. Otherwise, we will be available for general support starting at 7pm Pacific.
Thank you,
- Blue Box Group
Uplink and cross-connect network upgrades
On Saturday, July 9, 2011, starting at 11:00pm PDT (July 10, 06:00 UTC) and lasting for up to three hours, we will be upgrading both our internet connection uplinks, and several inter-switch cross-connects in our primary data center. As all of the links which are going to be upgraded are already redundant, except for a few brief pauses on the network as Spanning Tree reconverges after an uplink...
June 2011
1 post
Office Closed for Company Meeting
Our offices will be closed from 4:30pm Pacific - 6pm Pacific today for a company meeting. During this time, if you have an urgent issue, please escalate your ticket and our team will respond. Otherwise, we will be available for general support starting at 7pm Pacific.
Thank you,
- Blue Box Group
Major Maintenance Notification: Core Router...
On Saturday, June 4th starting at 11:00pm PDT (June 5, 06:00 UTC) and lasting for up to four hours we will be performing significant upgrade work on our core routers in our primary datacenter near Seattle, WA.
Due to the redundant nature of our core routing infrastructure, throughout this window our network will remain up and running and accessible to the internet. However there will be four...
May 2011
3 posts
Network Subnet Outage
On May 25th, at 14:24:33.410, the 67.214.215.1/24 subnet experienced a 10 minute outage (resolved at 14:34:05.581) as the result of networking routing issue with the subnet’s gateway address. Our team responded immediately upon notification at 14:27:40. The issue was corrected shortly there after. At this time, all systems are functioning normally. This outage affected a single subnet of...
Network Outage - May 6th, 2011
We have gotten a full report from our network provider as to the event that happened this morning between 04:10 and 04:20 PDT. They have confirmed that there was a full disruption to network connectivity between 04:12 and 04:13 PDT and that this will have affected traffic for a few moments around that timeframe. The cause of this outage was due to a faulty optical cable which is going to be...