From support@xmission.com  Wed Oct 29 17:05:04 MDT 2008
Date: Wed, 29 Oct 2008 17:05:04 MDT
From: XMission Support 
To: announce@xmission.com
Subject: DIRECT ANNOUNCE: Extensive UTOPIA Outage
Status: RO

Extensive UTOPIA Outage
-----------------------

Customers in many UTOPIA cities suffered a broad and extended outage of
service on October 27th from approximately 4:50 p.m. to 8:15 p.m.

The incident was caused by a dump truck cutting overhead fiber lines
within the UTOPIA network, initiating a severe networking problem.

Details
-------
Shortly before 5 p.m. on Monday afternoon, XMission technicians identified
a large UTOPIA network outage and immediately contacted UTOPIA about the
matter. UTOPIA engineers had difficulty troubleshooting the matter because
a fiber cut in a ring should not cause issues like this because MPLS
should ensure failover to the redundant path. 

Unfortunately, it was not determined that the secondary path was also
affected until FSG crews were sent out to troubleshoot in the field and
track down this issue. They discovered that a dump truck was unloading a
load of dirt by 4500 S 500 W and caught one of our aerial lines and caused
damage to several buffer tubes. They are currently working on repairing
this damage.  During the outage and with damage to both the primary and
secondary paths, UTOPIA's network engineers identified why the MPLS cloud
was unable to properly distribute XMission services through the UTOPIA
network. In order to resolve this issue they configured alternative MPLS
paths to restore services for XMission in the MPLS cloud.  Finally, they
are working on restoring the primary and secondary paths to restore full
redundancy on the UTOPIA network.

Conclusion
----------
XMission is working with UTOPIA to both improve their timeframe for
troubleshooting-to-resolution and vulnerabilities like a collapsed ring
where true redundancy is not ensured. We expect more from this
next-generation network and promise better service to our customers.

----------------------------------------------------------------------------
This is an XMission Direct Announcement.   Past announcements available at:
     WWW      - http://www.xmission.com/cgi-bin/announcements
     Homepage - http://home.xmission.com
     News     - xmission.announce
     FAQ      - http://www.xmission.com/help/misc/faqs/announcements.html