Zero IP Re-Numbering: VMware HCX Bulk Live Migration of 200+ Enterprise VMs

“Re-IPing 200 enterprise production virtual machines requires updating thousands of hardcoded application connection strings. Extended L2 network stretch avoids the nightmare.”

The Setup

In May 2018 at IBM, we designed a major datacenter evacuation for a enterprise financial client migrating from an aging on-premises VMware vSphere 6.0 cluster to an IBM Cloud VMware Solution (SDDC) environment powered by VMware NSX-V and HCX (Hybrid Cloud Extension).

The target environment contained 200+ mission-critical virtual machines running tier-1 database and middleware workloads. The client explicitly banned IP re-numbering due to hardcoded legacy configurations.

The Mess

To move live virtual machines across datacenters without changing IP addresses, we stretched on-premises VLAN subnets (10.150.20.0/24) into cloud NSX VXLAN overlays using VMware HCX Network Extension (L2 Extension).

During the initial bulk live migration batch, 40 VMs suffered sudden packet loss and dropped database connections midway through vMotion transfer:

[CRITICAL] 2018-05-14 03:22:15 UTC - VMware HCX L2 Extension Manager Log
[hcx-l2-ext-01] WARN  com.vmware.vnic.tunnel - WAN Tunnel packet drop detected on Appliance HCX-NE-IX-A1
[hcx-l2-ext-01] ERROR com.vmware.vmotion.net - vMotion Migration Failed for VM 'app-db-prod-04'
Reason: Extended L2 WAN MTU mismatch. Path MTU 1500 insufficient for VXLAN Encapsulated Packet (1572 bytes).
Result: vMotion Timeout (Heartbeat missed for 120 seconds). VM execution stunned.

The underlying WAN connection between the on-premises datacenter and IBM Cloud had an MTU limit of 1500 bytes. The HCX encapsulation added 72 bytes of tunnel overhead, causing MTU fragmentation and live vMotion stun failures.

The Solution

I resolved the encapsulation MTU mismatch and orchestrated a zero-downtime VMware HCX Bulk vMotion migration pipeline:

1. Jumbo Frame Path Optimization: Reconfigured on-premises edge routers and IBM Cloud Direct Link circuits to support MTU 9000 (Jumbo Frames) end-to-end.
2. HCX WAN Optimization & Compression: Enabled HCX WAN Optimization with deduplication and inline TCP acceleration.
3. Staggered Bulk Migration Waves: Grouped virtual machines into dependency-aware migration waves using HCX Mobility Groups.

# Verify End-to-End MTU 9000 Support across HCX L2 Appliance Handoff
ping -M do -s 8972 10.200.15.50

# Output:
# PING 10.200.15.50 (10.200.15.50) 8972(9000) bytes of data.
# 8980 bytes from 10.200.15.50: icmp_seq=1 ttl=64 time=1.42 ms
# 8980 bytes from 10.200.15.50: icmp_seq=2 ttl=64 time=1.38 ms
# OK - 0 Fragments, MTU 9000 Verified across Direct Link WAN.

# VMware HCX Mobility Group Migration Command
hcx-cli migration start-group --group-id MG-PROD-WAVE-01 --type BulkMigration --schedule "2018-05-20T01:00:00Z"
Result: 50 VMs scheduled for background delta sync + final zero-downtime cutover.

The Results

The HCX bulk live migration successfully evacuated the legacy datacenter with zero downtime:

• Workloads Migrated: Successfully migrated 200+ enterprise VMs across 4 migration waves.
• IP Re-Numbering Required: ZERO IP changes — 100% of subnets preserved.
• Application Downtime: 0 seconds downtime achieved across core database tier during live vMotion cutovers.

Key Takeaway

Always verify end-to-end Jumbo Frame MTU 9000 support across Direct Link WAN circuits before stretching Layer 2 networks with VMware HCX. Eliminating packet fragmentation guarantees smooth vMotion transfers without VM stun.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin