The BGP Steering Incident That Killed Our Low-Latency Guarantee

“The AI platform was ‘working’. Inference responses were just taking 340ms instead of 18ms. Nobody paged. Nobody noticed. The SLA was silent.”

The Setup

The cloud platform I work on runs latency-sensitive AI inference workloads over Azure ExpressRoute — private circuits connecting on-premises GPU infrastructure into Azure AI Services endpoints. The SLA is tight: P95 inference latency under 35ms. We have two circuits: a primary 10Gbps ExpressRoute at a Mumbai PoP and a secondary 1Gbps circuit via a Singapore PoP as backup.

BGP route control is how we ensure traffic prefers the Mumbai circuit. The mechanism: BGP Local Preference attribute. Higher is preferred. Primary circuit: 200. Secondary circuit: 100. Clean and simple.

The Change That Started It

A junior network engineer made a routine maintenance change — rotating the BGP session credentials for the secondary ExpressRoute circuit. Standard procedure, documented in the runbook.

The runbook was missing one step. After re-establishing the BGP session on the secondary circuit, the engineer did not verify the Local Preference values were re-applied.

The change control was approved. The BGP session came up. Traffic kept flowing. Monitoring showed green across all dashboards.

What the dashboards did not show: the secondary circuit had re-established with a Local Preference of 0 — the default. With Primary at 200 and Secondary at 0, traffic should still prefer Primary. It did.

For twelve hours.

The Mess

At 2:17 AM, a separate maintenance window on the primary circuit started — a planned firmware upgrade on the MSEE (Microsoft Enterprise Edge) router at the Mumbai PoP. The circuit went offline for forty minutes, exactly as expected. Traffic failed over to the secondary Singapore circuit, exactly as designed.

The primary came back at 3:01 AM. The BGP session re-established. Here is where the runbook gap became an incident.

When the primary re-established, it came up with its route advertisement including a Local Preference of 200 — correct. But the secondary circuit, which had been carrying traffic for forty minutes, was still advertising routes with Local Preference 0.

BGP route selection does not automatically re-evaluate running traffic flows on existing sessions when a better path appears. Traffic that was mid-session on the secondary stayed on the secondary. New flows went to the primary. But the long-lived AI inference TCP sessions — which persist for session affinity — kept routing through Singapore.

flowchart LR
    Client["Enterprise Client AI Workload"]
    HubVNet["Azure Hub VNet (Primary Region)"]

    ExpressRoutePrimary["Primary ExpressRoute (Mumbai)\nLocalPref: 200 (18ms)"]
    ExpressRouteSecondary["Secondary ExpressRoute (Singapore)\nLocalPref: 0 (340ms - STUCK SESSION)"]

    Client -->|New HTTP Sessions| ExpressRoutePrimary
    Client -->|Persistent Long-Lived Flows| ExpressRouteSecondary

    ExpressRoutePrimary --> HubVNet
    ExpressRouteSecondary --> HubVNet

    style ExpressRoutePrimary fill:#0f172a,stroke:#34d399,stroke-width:2px;
    style ExpressRouteSecondary fill:#0f172a,stroke:#f43f5e,stroke-width:2px;
    style HubVNet fill:#1e293b,stroke:#38bdf8,stroke-width:2px;

# BGP route table at 3:05 AM — observed anomaly
Network          Next Hop         LocPrf  Weight  Path
10.100.0.0/24   172.16.10.1       200      0      65001 (PRIMARY - correct)
10.100.0.0/24   172.16.20.1         0      0      65002 (SECONDARY - wrong LocPrf, still serving traffic)

The inference service was split-brained. New sessions: 18ms (Mumbai). Existing sessions: 340ms (Singapore). Session-level telemetry was not separated by circuit path. The aggregate P95 showed 89ms — above threshold, but not enough to trigger the alert. The alert was set for P95 > 200ms.

We found it 72 hours later when a recruiter’s AI twin demo went visibly sluggish during a live call. The response latency was captured in application logs. That is a terrible way to detect an SLA breach.

---

The Fix: Three Changes, One Architecture Decision

Fix 1: Route Policy Baseline Verification Script

# scripts/verify_bgp_baseline.py
# Run after every BGP session change — fails if Local Pref deviance detected

import subprocess
import json

EXPECTED = {
    "primary_circuit": {"local_pref": 200, "peer": "172.16.10.1"},
    "secondary_circuit": {"local_pref": 100, "peer": "172.16.20.1"}
}

def get_bgp_routes():
    # Using Azure Network Watcher BGP peer API
    result = subprocess.run([
        "az", "network", "express-route", "list-route-tables",
        "--resource-group", "rg-connectivity-prod",
        "--name", "er-circuit-primary",
        "--peering-name", "AzurePrivatePeering",
        "--path", "primary",
        "--output", "json"
    ], capture_output=True, text=True)
    return json.loads(result.stdout)

def verify_local_pref(routes):
    for route in routes.get("value", []):
        if route.get("localPref") != EXPECTED["primary_circuit"]["local_pref"]:
            raise ValueError(
                f"BASELINE_VIOLATION: Route {route['network']} has "
                f"LocalPref={route.get('localPref')} expected=200"
            )

if __name__ == "__main__":
    routes = get_bgp_routes()
    verify_local_pref(routes)
    print("✅ BGP baseline verified — Local Preference values correct")

This now runs as a post-change hook in every ExpressRoute maintenance runbook.

Fix 2: Per-Circuit Latency Telemetry Separation

We instrumented the inference gateway to tag all requests with the egress circuit identifier via a custom HTTP header injected by the Azure Route Server policy. Azure Monitor Log Analytics now has:

// Latency split by ExpressRoute circuit path
AzureDiagnostics
| where Category == "InferenceGatewayMetrics"
| summarize
    P95Latency = percentile(DurationMs, 95),
    RequestCount = count()
  by bin(TimeGenerated, 5m), tostring(properties.CircuitPath)
| render timechart

Fix 3: Lowered SLA Alert Threshold to P95 > 50ms with Per-Circuit Breakdown

The 200ms alert was set for user-visible degradation. We now alert at 50ms on any single circuit path — well below user-visible threshold, catches BGP path anomalies before they aggregate into something ugly.

Post-Incident Finding

The root cause was not the engineer’s mistake. The root cause was a runbook that did not include a verification step and an alerting strategy that aggregated across all paths rather than monitoring each independently.

BGP route control is not a set-and-forget configuration. Every session re-establishment is a potential Local Preference reset. Every circuit failover is a potential traffic split. The monitoring needs to match the blast radius.

Key Takeaway

If your ExpressRoute failover strategy relies on BGP Local Preference, add a post-session-change verification step to every runbook. Not recommended — mandatory. BGP will not warn you when it silently demotes a circuit. You find out when someone’s demo goes sluggish.

Latency SLAs measured at the aggregate level will always hide per-path degradation. Separate your telemetry by circuit path before your next planned maintenance window.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin