When BGP Local-Pref Steered Us into a Routing Loop

“Everyone told me to filter dynamic prefixes at the edge. Everyone was wrong. We should have filtered our own advertisements at the transit boundary.”

The Setup

It was June 2012. We were managing two Tier-3 data centers during a high-stakes multi-homed provider migration.

Our core edge was anchored by a pair of Cisco ASR 1002 routers, connected to two upstream transit ASNs: AS2389 (Primary) and AS4608 (Secondary). We owned a public /22 IPv4 address block and were running multi-homed BGP peering sessions to achieve automated link failover.

graph TD
    User[Real Visitor] --> PrimaryAS[AS2389 - Primary Transit]
    User --> BackupAS[AS4608 - Secondary Transit]
    PrimaryAS --> Edge1[Cisco ASR 1002 - A]
    BackupAS --> Edge2[Cisco ASR 1002 - B]
    Edge1 <--> Core[Datacenter Core Switch]
    Edge2 <--> Core

The Mess

The ticket was routine: shift 70% of inbound traffic from AS4608 to AS2389 to meet monthly commitment quotas. I executed a standard BGP route-map modification on Edge-B to prefix-list match our /22 block and append our ASN three times (AS-PATH prepending).

Five seconds after committing, the monitoring console went completely dark.

%BGP-3-NOTIFICATION: sent to neighbor 203.0.113.1 3/2 (update error: path attribute collision)
%LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet0/1/0, changed state to down

The first wrong diagnosis was an upstream fiber cut. I spent 15 frantic minutes calling the AS4608 NOC trying to verify physical carrier drops. They insisted their links were clean, but telemetry showed our routes bouncing between ASNs every 900ms.

The moment of truth came when I ran a packet capture on the ASR console and saw our own outgoing packets bouncing directly back to us from the secondary transit provider, looping continuously until TTL expired.

# tcpdump -i te0/1/0 -vvv 'ip proto 89 or port 179'
22:05:14.912 BGP: 203.0.113.1 > 203.0.113.2: UPDATE (179 bytes)
  Path Attribute: AS-PATH (64512 64512 64512 64512)
  Originating Prefix: 198.51.100.0/22
22:05:15.812 BGP: 203.0.113.2 > 203.0.113.1: UPDATE (179 bytes)
  Path Attribute: AS-PATH (64512 64512 64512 64512 64512)

The prepended path was looping right back into our second edge router because we forgot to filter local BGP preferences on the internal connection between the ASR pairs.

The Solution

We immediately tore down the active BGP peering sessions to break the looping traffic. I then rewrote the edge BGP configuration blocks to enforce strict local-preference metrics on Edge-A (Primary, local-preference 150) and Edge-B (Secondary, local-preference 100), ensuring symmetric internal routing.

Here is the exact Cisco ASR IOS configuration applied to reconcile and secure the BGP edge boundary:

! Configure strict BGP path selection policy
route-map PREPEND-OUT permit 10
 match ip address prefix-list OUR-PREFIXES
 set as-path prepend 64512 64512 64512
!
route-map LOCAL-PREF-IN permit 10
 set local-preference 150
!
router bgp 64512
 bgp log-neighbor-changes
 neighbor 203.0.113.1 remote-as 2389
 neighbor 203.0.113.1 route-map LOCAL-PREF-IN in
 neighbor 203.0.113.1 route-map PREPEND-OUT out

Applying this configuration aligned our inbound metrics and instantly stabilized the routes. Round-trip times dropped from infinite timeouts back to a flat 12ms.

Key Takeaway

Never trust prepending alone to steer incoming BGP traffic in multi-homed systems. If you do not lock down internal route preferences, the internet will find the shortest logical path and steer your packages straight into a local routing loop.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin