From support@xmission.com  Tue Aug 04 14:44:02 MDT 2009
Date: Tue, 04 Aug 2009 14:44:02 MDT
From: XMission Support 
To: announce@xmission.com
Subject: ANNOUNCEMENT: UTOPIA and DSL Outage
Status: RO

UTOPIA and DSL Outage
---------------------
XMission's UTOPIA and DSL customers experienced an outage in the middle of
the night on Tuesday, August 4th due to a problem with the router that
handles connections for these customers.

Details
-------
XMission uses the Juniper ERX, a carrier class router, to service our DSL
and UTOPIA customers. The problem initially only affected UTOPIA customers
but eventually caused problems for DSL customers as well. As a result, the
outage windows were slightly different for both groups:

DSL outage window:    4:38 a.m. - 5:15 a.m.

UTOPIA outage window: 3:17 a.m. - 4:11 a.m.
UTOPIA outage window: 4:38 a.m. - 5:30 a.m.

For those interested, here is an overview of the events last night.  We
had a failure of one of the I/O backplane slots on the ERX.  After
multiple attempts to bring the line card back online, the line card was
moved to a different slot.  This functioned for approximately 40 minutes,
at which time the OS started to degrade in usability.  Within a few
minutes the standby SRP automatically took over, but unfortunately failed
to bring the ERX back to working order.  At this point, both UTOPIA and
DSL customers lost routing.  Our network administrator determined a full
power cycle was necessary, which brought the ERX back to full working
order.

A full power cycle was not performed at the outset of problems because it
can take up to 30 minutes for the ERX to come back online and begin to
reconnect all customers.

Conclusion
----------
We are investigating the cause for this outage but suspect it was a bug.
We have already submitted a ticket to Juniper and are evaluating new
versions of JunOS.  Our ERX has been very stable for nearly 3 years so we
have every confidence its reliability.


Faster connections make for happier students.  Upgrade in time for back to
school and enjoy XMission's summer rates!  Be sure to ask about our
pay-in-advance discounts when you call.

------------------------------------------------------------------------------
This has been an XMission Announcement.   Past announcements available at: 
     WWW      - http://www.xmission.com/about/announcements
     Home     - http://www.xmission.com
     Portal   - http://home.xmission.com
     Status   - http://stats.xmission.com/netstatus