The Azure ExpressRoute BGP Steering Incident: 3 Weeks of Asymmetric Drops

“Everyone told me our Azure ExpressRoute failover was automated. Until a £40,000 asymmetric packet drop proved that ‘automated’ meant ‘untested’.”

In my current role leading Enterprise Cloud Architecture, we deployed dual 10 Gbps ExpressRoute circuits linking our on-premises core network to an Azure Enterprise Landing Zone across two geographically distinct peering locations (UK South and UK West).

The objective was straightforward: Primary-Active pathing through UK South, with UK West acting strictly as a passive standby circuit for disaster recovery.

---

The Setup

Both circuits terminated on a redundant pair of edge routers running MP-BGP dual-stack peering into Azure Virtual Network Gateways (ExpressRoute Direct).

To enforce deterministic traffic flow, all outbound traffic from on-premises to Azure was supposed to prefer the UK South circuit, while inbound traffic from Azure to on-premises was governed by AS-Path Prepending on the UK West circuit.

On paper, the architecture was rock solid. In practice, packet drops began happening silently within 48 hours of go-live.

---

The Mess

The initial symptoms were chaotic. Users reported intermittent HTTP 504 timeouts on internal SAP workloads, while SSH sessions to cloud VMs froze randomly after 30 seconds of inactivity.

The firewall team immediately pointed fingers at the Palo Alto NGFW cluster, assuming session table exhaustion. We spent 6 days tuning TCP keep-alive timeouts and increasing firewall state memory. That made it worse—bypassing stateful TCP checks allowed half-open sessions to linger, consuming even more memory.

The breakthrough happened at 2:15 AM during a live packet capture on the edge boundary.

# Executing BGP route table inspection on primary edge router
show ip bgp neighbors 192.168.100.1 received-routes | include 10.240.0.0

# Result: Equal-Cost Multi-Pathing (ECMP) active on outbound routes
*> 10.240.0.0/16     192.168.100.1          0    100      0 65515 i
*                    192.168.102.1          0    100      0 65515 i
# BOTH paths had identical local-preference (100)!

Outbound packets were leaving through UK South, but returning packets were being routed back via UK West because the BGP Local Preference attribute was missing in the Terraform HCL module definitions.

The firewall on UK West received TCP ACK packets without ever seeing the initial SYN packet. As a stateful firewall should, it dropped them instantly.

Worse, two automated Checkov static analysis scans in our Azure DevOps CI/CD pipeline had passed cleanly because our pipeline lacked a custom policy to validate BGP attribute variables in Infrastructure-as-Code.

---

The Solution

We fixed the outage in two phases: emergency BGP route-map injection on the live edge routers, followed by codifying strict Checkov IaC guardrails to prevent un-anchored route-maps from ever reaching production again.

First, we injected an explicit higher BGP Local Preference (200) for all prefixes arriving via the UK South circuit:

# Immediate hotfix: Cisco IOS-XE BGP Local Preference override
router bgp 65001
 address-family ipv4
  neighbor 192.168.100.1 route-map MAP-EXPRESSROUTE-PRIMARY-IN in
!
route-map MAP-EXPRESSROUTE-PRIMARY-IN permit 10
 set local-preference 200
# Prefixes via UK South now decisively win BGP path selection

Next, we added custom Checkov Python guardrails into our Terraform repository (policies/checkov/expressroute_bgp.py) to enforce that every ExpressRoute circuit module explicitly defines local_preference:

# Custom Checkov IaC Policy: Enforce BGP Local Preference on ExpressRoute Modules
from checkov.common.models.enums import CheckResult, CheckCategories
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck

class ExpressRouteBgpSteeringCheck(BaseResourceCheck):
    def __init__(self):
        name = "Ensure ExpressRoute BGP circuits explicitly set Local Preference"
        id = "CKV_AZURE_CUSTOM_01"
        supported_resources = ['azurerm_express_route_circuit_peering']
        categories = [CheckCategories.NETWORKING]
        super().__init__(name=name, id=id, categories=categories, supported_resources=supported_resources)

    def scan_resource_conf(self, conf):
        if 'peer_weight' in conf or 'local_preference' in conf:
            return CheckResult.PASSED
        return CheckResult.FAILED

check = ExpressRouteBgpSteeringCheck()

---

Key Takeaway

[!IMPORTANT] Asymmetric routing is not a cloud network failure; it is an IaC policy failure. If your CI/CD pipeline tests Terraform syntax but doesn’t validate BGP path selection attributes, your failover is a gamble.

The Verdict

Never trust vendor defaults for multi-region hybrid WAN links. Always enforce Local Preference (200 vs 100) for outbound traffic steering, combine it with AS-Path Prepending (3x) for inbound returns, and lock it down in CI/CD with custom Checkov policies before running terraform apply.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin