From support@xmission.com  Mon Sep 14 16:23:25 MDT 2009
Date: Mon, 14 Sep 2009 16:23:25 MDT
From: XMission Support 
To: announce@xmission.com
Subject: ANNOUNCEMENT: ERX Routing Issue
Status: RO

ERX Routing Issue
-----------------

On Friday, September 11th from 11:20 a.m. until 12:13 p.m., some XMission
DSL and UTOPIA customers experienced internal routing issues between other
XMission services (Colocation, Web Hosting, Mail Servers, etc.).  

Although Internet traffic was unaffected, XMission does have an on site
Akamai cluster, which would have been unreachable by DSL and UTOPIA
customers during this time frame.

Details
-------
The BGP RPD on the router that connects our DSL and UTOPIA customers, a
Juniper ERX, stopped processing received routes correctly from our core
routers.  This was first noticed when an iBGP route from one of our other
access routers was not being accepted on the ERX.  Further inspection
revealed the ERX was rejecting any route which didn't have a
local-preference of 100.  Other than static routes for our aggregate
supernets, all internal XMission routes carry a local-pref greater than
100.  As the ERX serves the vast majority of internal routes, and to
minimize iBGP table bloat, the ERX also has a few aggregate static routes
set up.  With the ERX having a static route for 166.70.0.0/16, and no
longer accepting a route example of 166.70.238.0/29, traffic would be
dropped.  Fortunately the ERX is originated a default route, so if no
local route existed, traffic would pass unaffected into the core routers.

After a number of attempts to restore the RPD on one of the BGP sessions,
including soft/hard BGP resets, shutting, and minimizing received routes,
we were successful in restoring full service.  We are currently working
with Juniper to diagnose why the issue happened in the first place, and
why the final solution corrected the issue (specific JUNOSe command).

Conclusion
----------
Luckily, the number of affected customers was relatively low but XMission
takes reliability and disclosure seriously so we decided to notify all of
our customers. Please note that it might have appeared to some customers
that their colocated servers were unavailable to the world but only
XMission DSL and UTOPIA customers were affected.

Customers interested in notices regarding scheduled maintenance and
outages can refer to our system status page, where you can also sign up
for that mailing list:
  http://stats.xmission.com/netstatus/


Traveling for business? Take XMission's Hosted Email along for the ride!
Have access to this full collaboration suite - including contact lists,
calendars, and much more - from any Internet connection in the world!
------------------------------------------------------------------------------
This has been an XMission Announcement.   Past announcements available at: 
     WWW      - http://www.xmission.com/about/announcements
     Home     - http://www.xmission.com
     Portal   - http://home.xmission.com
     Status   - http://stats.xmission.com/netstatus