Blue Box System Status

Upstream (Cogent) routing issue resolved

Fri, 18 May 2012 15:31:10 -0400

(Update to the web only post)

Bluebox along with our direct upstream providers have isolated out our
exposure surrounding traffic crossing Cogent between 11:25am —> 11:35am
(pacific). Neither Buebox nor our direct providers utilize Cogent directly
meaning that the impact was minimal during this event.

From our graphing we witnessed between an 8 to 10 percent drop of inbound
traffic while Cogent was having severe packet loss to AT&T and others. The
major backbone players seem to be stable at the moment with some loss
between verizon & Level 3, but not nearly at the scale Cogent witnessed.

All systems are and remain in full functioning order.

Thank You,

BlueBox team

(previous web only post below)

Upstream Routing Issue

Fri, 18 May 2012 14:58:40 -0400

Currently we are investigating an upstream routing issue in which Cogent is
dropping vast amounts of traffic. BlueBox does not utilize Cogent as a
direct provider, however some users trying to reach BlueBox (or any other
sites) via Cogent are likely to encounter packetloss. We are taking
corrective action on our side to minimize impact from this.

Impact appears to be corrected now and appeared to have impacted roughly
10% of incoming traffic to our Seattle Primary Datacenter for a duration of
roughly 10 minutes.

(Web Post Only)

Re: lb04 dropping some packets

Wed, 02 May 2012 03:37:34 -0400

The lb04 issue has been resolved. It was tracked down to one instance
which was seeing unusual traffic patterns. We’ve resolved this and are
continuing to monitor the site.

(Web Only Site)

lb04 dropping some packets

Wed, 02 May 2012 03:06:12 -0400

One load balancer (lb04) is currently experiencing some packet drops.
No other load balancers are affected at this time.

We’re investigating and should have a resolution shortly.

Currently only a few sites are affected but we will keep monitoring
to see if it gets worse - should that happen we’ll fail over to the
redundant load balancer, but we will update you all if that is
necessary.

(Web Only Post)

Temporary Upstream Network Issue [resolved]

Sun, 29 Apr 2012 04:25:37 -0400

One of our upstream network providers experienced a network issue at
about 12:39am PDT (GMT -7) this morning, causing a BGP change.
Customers may have seen downtime lasting a 1 - 3 minutes during this
time. No residual network issues have been noticed, however the
upstream provider is working on hardware at this time, which may
result in one more BGP change when their work has finished. Please
open a ticket with Blue Box if you have any questions.

(web only post)

DNS issues [resolved]

Fri, 27 Apr 2012 09:54:25 -0400

Our primary DNS resolver used within Blue Box Group has been experiencing some issues which resulted in a disruption of services for some customers. While the secondary resolver should become active in such cases this did not happen for some customers. We did get the primary resolver running so everything should be functional again. Please let us know if your site is still experiencing any issues as a result of this DNS disruption.

Datacenter Maintenance Tonight - Seattle Primary DC

Thu, 12 Apr 2012 03:11:06 -0400

Maintenance has been completed, was non-intrusive to all services and all
areas of focus were completed and 100% successful.

(Web only post)

Datacenter Maintenance Tonight - Seattle Primary DC

Wed, 11 Apr 2012 21:07:38 -0400

Tonight starting at 11:00 pm Pacific we will be performing some physical
maintenance in one area of our Primary Seattle datacenter location. This
maintenance should be non-intrusive to all services and no impact is
expected. Estimated duration is 2 hours, 30 minutes and should be
completed no later then 02:00 am Pacific.

(Web post only)

Network Maintenance Complete

Thu, 05 Apr 2012 03:15:26 -0400

Maintenance has been completed, was non-intrusive to all services and all
areas of focus were completed and 100% successful.

(Web only post)

Network Maintenance Tonight

Wed, 04 Apr 2012 20:09:55 -0400

Tonight at 11:00pm PDT (06:00 UTC) we will be replacing a redundant stacked
device in our distribution layer which has shown to be unstable. No packet
loss, latency increases or other problems are anticipated as a result of
this work. However, as it does affect one of our distribution layers of
our networking gear the potential of a brief interruption of service does
exist.
Please note that this upgrade is to replace one device in a redundant
stack, we do not anticipate any issues in performing this work.

(Web only post)

Intermittent network issues

Mon, 02 Apr 2012 17:09:00 -0400

We’re currently getting notifications of some intermittent network issues. It seems limited to a small set of customers. We’re currently investigating this and will be able to provide an update soon.

Update 17:00EST: This issue seems to be related to a long haul provider upstream from our network. We have checked our networks and have not found any issues.

If you are still seeing issues, can you provide a traceroute to us which will assist us in determining which providers are having issues.

(Web only post)

Intermittent phone service issues

Tue, 27 Mar 2012 17:53:24 -0400

Our phone service is experiencing some issues and your contact with support
may be interrupted, If you need to contact support, please email us or
contact us in customer chat.

Update: Phones are now working again. Total outage time was about 8 minutes

(Web Only Post)

Scheduled Maintenance on Sunday, April 1st 23:00 to April 2nd 0:00 PDT

Mon, 26 Mar 2012 13:20:32 -0400

This is an alert regarding scheduled maintenance for this coming weekend.

Our transport provider will be preforming preventative maintenance on our transport circuit between our Tukwilla data center and our Seattle backup site. During this maintenance connectivity between the two sites will be interrupted.

Maintenance Window: Sunday, April 1st 23:00 to April 2nd 0:00 PDT

Internet connectivity to our Tukwilla and Ashburn data centers will not be affected and our Seattle backup site will fail over to its backup Internet connection. Servers affected at the backup site have IP addresses starting with 208.85.149.xxx, other servers will not be affected.

If you are running database replication or other sensitive applications between servers at our Tukwilla data center and backup site, we recommend stopping database replication during the window.

If you need assistance or have any questions please contact us at support@bluebox.net.

(web only post)

Network uplink upgrade [completed]

Thu, 22 Mar 2012 03:36:44 -0400

The maintenance has been successfully completed. Everything went according to plan. If you encounter any connectivity issues please let us know via our support system.

(Web only post)

Network uplink upgrade tonight

Thu, 22 Mar 2012 00:29:19 -0400

Tonight at 10:00pm PDT (05:00 UTC) we will be upgrading the capacity of one of our upstream links in order to stay ahead of growth. As part of this upgrade, we will be gracefully shutting down the BGP session with this peer to allow traffic to naturally abate before the work is performed. Because of this, no packet loss, latency increases or other problems are anticipated as a result of this work. However, as it does affect our core networking gear, an increased risk of an outage does exist.

Please note that while this upgrade will greatly increase our capacity, this work does not yet address all the factors which lead to last Sunday evening’s and Wednesday morning’s network problems. (Though this is a step in that process.)

(Web only post)

Network Event

Wed, 21 Mar 2012 07:51:00 -0400

We are investigating reports of a network event and will post more information here shortly.

If you are experiencing any issues, please open an urgent support ticket, or call us at 800-613-4305 and leave us an urgent voicemail. Our team will get back to you immediately.

Update 5:00am - At this point in time, we believe this to be related to a BGP event with upstream providers. The network appears stable at this time and we are now researching what happened.

Update 5:05am - Monitoring indicates a disruption based on which BGP route a user was taking connecting to our network. It appears total disruption was between 4 and 9 minutes and that connectivity was restored at 4:59am. The network continues to function normally and our team continues to investigate.

Update 5:15am - The network continues to function normally and our team continues to investigate. If you are experiencing issues, please contact support at support@bluebox.net.

Update 5:30am - This event appears related to a single upstream connection that bounced, causing traffic coming over that link to be interrupted and the BGP session to be reset.

Update 5:45am - Attached is the associated alerting from Pingdom indicating the duration of the event. Pingdom reports this beginning at 03/21/2012 4:53:32AM pacific, and clearing at 03/21/2012 4:55:34AM.

Update 6am - We’ve confirmed this event was the result of a flap in a connection to one of our redundant upstreams. This is the same upstream that experienced issues on Sunday night. The issue occurred on the remote end of the gear, not controlled by BBG. We are awaiting final details from that provider as to what happened from that vendor and will update this post later today once we have that information.

The total visible impact would have been an interruption in routing for a portion of the customer base transiting via this link for a period of aprox 3-5 minutes. External reporting may indicate a slightly longer outage based on the polling window of the service utilized.

Based on these last two events, Blue Box is evaluating options of turning up a replacement provider and discontinuing existing service with the provider in question. We had begun that process on Monday based on the event Sunday evening, but we will now escalate that activity. Additionally, our networking team will be evaluating options at reducing our exposure to these types of events prior to our transition to the different transit provider.

Update 2:54pm - Upstream provider confirmed that the disruption this morning was caused by a failed line card, which caused a systematic cycle on one side of their Border Router Infrastructure as well as a hung CPU on the other side. We are expediting steps to isolate Blue Box from this provider in a effort to eliminate our exposure to possible future event.

(Web only post)

BGP Re-convergence Event

Mon, 19 Mar 2012 01:21:00 -0400

This evening, starting at 10:00pm Pacific, one of our upstream connectivity partners experienced a failure in their routing infrastructure. BGP has re-routed that traffic over our redundant connections and all traffic should be flowing normally at this time. Please open an urgent support ticket if you are experiencing any residual issues.

Update 10:29pm - We’ve learned from our provider that this was an unplanned emergency maintenance event that started at 10:00pm Pacific. Our networking team was not notified of this event in advance. Traffic continues to flow over our redundant connections normally.

(Web only post)

Network Issues

Tue, 13 Mar 2012 07:11:00 -0400

We have been made aware about a brief period of network related issues affecting some customers. The issues have now been cleared and everything should be working as expected again. We are still investigating the root cause.

Update: The above issue caused some intermittent packet loss for around 10 minutes between 3:45 and 3:55am PDT (10:45-10:55 UTC). The issue was traced back to a compromised client machine on the network attempting to DOS another host on the internet. We have shut this machine down and are working with the client in question on clean-up.

We will also be rolling a slight network configuration change forward later today which will make it much more difficult for a single compromised machine on our network to cause problems for any of our other clients on our network.

(Web only post)

block-lb02 to be returned to production

Wed, 07 Mar 2012 18:42:49 -0500

As part of Monday’s power issue, one of our Blocks Load Balancer machines (block-lb02) was removed from production when it lost power. Its HA partner automatically took over quickly thereafter. However, to minimize the potential impact on customers during peak hours we kept block-lb02 out of a production role until we were sure the power situation had been addressed.

Tonight shortly after 11:00pm PST (07:00 UTC) we will be returning block-lb02 to a production role. When we do this, block-lb02 will be taking over load balancer services from its standby partner (block-lb01). During this transition, some customers with load balanced services may experience up to 2 minutes of interruption as the services transition from block-lb01 to block-lb02.

(Web only post)

All Systems Normal

Mon, 05 Mar 2012 18:29:00 -0500

All customers on vsh222 are now back online. We will be following up with all affected customers directly. If you are experiencing issues, please contact support at support@bluebox.net.