From support@xmission.com  Fri Dec 12 17:16:02 MST 2008
Date: Fri, 12 Dec 2008 17:16:02 MST
From: XMission Support 
To: announce@xmission.com
Subject: DIRECT ANNOUNCE: DSL Issues Resolved
Status: RO

DSL Issues Resolved
-------------------

Starting at approximately 4:50 a.m. on Thursday, December 11th, about
1,000 of our DSL customers in the southern end of Salt Lake valley
experienced problems with their DSL service.  Most affected customers saw
speed and latency issues. 

The problem was caused by a saturated network connection inside Qwest's
network after a router failed that serviced the Murray area. The issue
lasted far too long because Qwest thought they had fixed the problem. At
this time, the majority of customers have been fixed but we are working
with Qwest to try to get all DSL customers taken care of before the end of
the day. 

Details
-------
As we understand it, the issue started when a router inside Qwest's
network failed Thursday morning and DSL traffic in Murray and surrounding
areas was redirected from multiple trunks to a single failover OC-12,
causing saturation.

Overnight, the saturated OC-12 was partially offloaded onto alternative
trunking to relieve the capacity trouble.  Most users seem to have been
repaired by this. Before doing this, the only fix was to have Qwest do a
retranslation on the customer's DSL line but that is an arduous process.
The retranslation moves the customer off the overloaded trunk and over to
a different one. Yesterday, we sent them 654 phone numbers we suspected
were affected as we noticed those customers went down at 4:50 a.m. when
Qwest's router failed.

Conclusion
----------
While we believe that Qwest has finally resolved this issue for most
customers, please contact our technical support department if you live in
the southern part of Salt Lake valley and believe you are still affected.

This was the strangest Qwest DSL issue we have ever seen, not only because
we have never experienced a problem like this before but because Qwest
struggled to properly identify the source and scope of the matter
internally. As with most large corporations, various departments don't
appear to communicate well and their front line support technicians were
left just scratching their heads when some of you called them.
Unfortunately, we do not have reasonable access to Qwest's engineers
either so we only started to make progress once we lucked upon a
particularly helpful upper tier support person at Qwest who was willing to
work through this matter with us. Today, we barked our way up high enough
to get further help and details.

----------------------------------------------------------------------------
This is an XMission Direct Announcement.   
     WWW      - http://www.xmission.com/about/announcements
     Portal   - http://home.xmission.com
     Status   - http://stats.xmission.com/netstat