From support@xmission.com  Wed May 13 13:37:35 MDT 2009
Date: Wed, 13 May 2009 13:37:35 MDT
From: XMission Support 
To: announce@xmission.com
Subject: ANNOUNCEMENT: XO Equipment Failure Affected Dial-up, T1, and XMission Phone Lines
Status: RO

XO Equipment Failure Affected Dial-up, T1, and XMission Phone Lines
-------------------------------------------------------------------

Late at night, on May 12th, XO Communication's OC48 hardware at XMission
failed while an XO technician was on-site investigating a potential issue
with the equipment. This disabled XMission's office phone lines, most of
our dial-up services, and T1 customers that use XO for transport.

XMission network administrators were in shortly after the incident began.
Within 40 minutes, the equipment was replaced and all XO circuits were
back in service.

Details
-------
The outage lasted from 10:54:40 p.m. until 11:33:51 p.m. on May 12th.

XO had an emergency line card they swapped to repair both TCC blades (Time
Controller Card) in their SONET switch. Since the TCCs were already
degraded, when the XO technician swapped out the standby TCC, the active
TCC blade failed while it was booting.  At this point, both backplane
controllers failed hard, taking down the entire switch, and breaking the
local database and configuration.

As the new TCC blades were running a version of the OS that was not
compatible with the other line cards, the XO tech had to downgrade them,
and then recover a backup of the database to finally bring the switch
online.  This caused the majority of the downtime.

Conclusion
----------
While this outage was critical, most XMission customers were not affected
due to the redundancy of our network.

This was unscheduled emergency maintenance by XO which did not allow for
advanced notice on the netstatus list. However, once the outage occurred
the netstatus information was immediately updated. If you are not already
signed up, please subscribe to the netstatus mailing list via this link:
http://stats.xmission.com/netstatus/
  
 
Keep up with XMission's latest via Twitter. Ask us questions, or let us
know what we can do to improve your service! http://twitter.com/xmission

You can also become an XMission fan on Facebook:
http://www.facebook.com/pages/Salt-Lake-City-UT/XMission/6155383019

------------------------------------------------------------------------------
This has been an XMission Announcement.   Past announcements available at: 
     WWW      - http://www.xmission.com/about/announcements
     Home     - http://www.xmission.com
     Portal   - http://home.xmission.com
     Status   - http://stats.xmission.com/netstatus