GENEVE vs VXLAN: Why We Re-Engineered 400 Hypervisors During an Live Cloud Migration

“Everyone assumes overlay encapsulation is just 50 bytes of extra overhead. Until a core switch silently drops 1600-byte packets because a vendor hardcoded MTU at 1550.”

---

The Setup

In 2018 at IBM Cloud, we were tasked with migrating a multi-tenant enterprise SDDC environment from VMware NSX-V to NSX-T.

The legacy infrastructure relied on VXLAN — the workhorse of 2010s software-defined networking. VXLAN uses a fixed 8-byte header with a 24-bit VXLAN Network Identifier (VNI). It worked fine for standard VLAN bridging.

+-------------------------------------------------------------------+
| Outer IP (20B) | UDP (8B) | VXLAN Header (8B) | Inner Ethernet... |
+-------------------------------------------------------------------+

Our goal was to transition 400+ ESXi hosts and 2,500 workload VMs to NSX-T utilizing GENEVE (Generic Network Virtualization Encapsulation, RFC 8926).

GENEVE replaced fixed headers with flexible Type-Length-Value (TLV) metadata fields, enabling micro-segmentation tags and telemetry data to travel directly inside the packet header.

---

The Mess

The migration playbook looked straightforward on paper. We set up VMware HCX bulk migration pipelines and configured cross-cluster Layer-2 extensions.

Then the midnight change window hit.

[00:14:22] HCX L2 Extension Established: vni-5001 <-> geneve-10001
[00:19:05] WARN: vMotion sync stalled at 88% for db-cluster-prod-02
[00:22:11] ALERT: Arista HW-VTEP OVSDB connection lost on Spine-01
[00:24:00] ERROR: Packet drop detected. Path MTU discovery failing for payload > 1544 bytes

Workload migrations froze mid-flight. Application latency spiked by 400ms across hybrid tunnels.

# ICMP ping sweep test across the overlay tunnel
ping -M do -s 1572 10.240.12.105

# Output:
# From 10.240.12.1 Packet needs to be fragmented but DF set.

Here is what failed first:

• The MTU Trap: VXLAN required an IP MTU of 1550 bytes (1500 byte payload + 50 byte encapsulation). But GENEVE’s dynamic TLV headers push encapsulation overhead up to 80 bytes.
• The Hardware VTEP Bug: Our ToR (Top of Rack) Arista switches were acting as Hardware VTEPs via OVSDB. The switch firmware inspected the fixed 8-byte offset expecting VXLAN. When it hit GENEVE’s variable-length TLV options, it misclassified the packet as malformed UDP and dropped it in ASIC hardware.
• Path MTU Discovery (PMTUD) Collapse: Firewalls between data center zones were dropping ICMP Type 3 Code 4 (“Fragmentation Needed”) packets, breaking PMTUD silently.

We were sitting in the NOC with 2,500 production VMs hanging on extended Layer-2 bridges while Arista hardware switches dropped packets every time NSX-T attached trace tags.

---

The Solution

We had to resolve the encapsulation mismatch live without rolling back 48 hours of maintenance preparation.

---

Step 1: Standardize Global Jumbo Frames to 1700 Bytes

We elevated physical fabric MTU across all spine switches, leaf switches, and vSphere Distributed Switches (vDS) from 1550 to 1700 bytes.

# Verify physical ESXi VMkernel interface MTU setting
esxcli network ip interface list | grep -E "(Name|MTU)"

# Output:
# Name: vmk10 (NSX TEP)
# MTU: 1700

Step 2: Custom Netmiko Python Script for Switch Port Realignment

We executed an automated Python script using Netmiko to sweep all Arista and Cisco Nexus switch interfaces, updating system MTU and setting OVSDB encapsulation to GENEVE explicitly.

# scripts/fix_tep_mtu.py - Netmiko fabric MTU update
from netmiko import ConnectHandler

def update_fabric_mtu(device_info):
    connection = ConnectHandler(**device_info)
    commands = [
        "system mtu 1700",
        "interface Ethernet1/1-48",
        "mtu 1700",
        "cvx",
        "no service",
        "service ovsdb",
    ]
    output = connection.send_config_set(commands)
    connection.save_config()
    print(f"[{device_info['host']}] Configured MTU 1700 & OVSDB GENEVE mode")

# Applied across 32 fabric switches in parallel

Step 3: Re-Engineered Layer-2 Extension Flow

We re-architected the encapsulation pipeline so that legacy VXLAN traffic bridged to GENEVE via dedicated NSX Edge Nodes rather than Top-of-Rack hardware switches.

+------------------+         +--------------------+         +------------------+
| Legacy NSX-V VM  | ------> |  NSX Edge Bridge   | ------> |   NSX-T Pod VM   |
|  (VXLAN VNI)     |         | (Translates Header)|         |  (GENEVE VNI)    |
+------------------+         +--------------------+         +------------------+
                                MTU 1700 Fabric

---

Key Takeaway

Never assume overlay protocols are transparent to physical hardware. GENEVE’s variable-length TLV header is superior for telemetry and security context, but legacy switch ASICs will drop packets if they expect fixed VXLAN offsets. Always configure physical fabric MTU to at least 1700 bytes before deploying NSX-T.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin