“The enterprise migration requirement was non-negotiable: ‘Migrate 500+ workloads to the cloud, but do not change a single IP address or subnet default gateway.’”

Executive Summary

In 2018, during my tenure as a VMware NSX Consultant at IBM, a major manufacturing client initiated a full datacenter evacuation into IBM Cloud (SoftLayer).

The primary migration blocker was a legacy Oracle RAC database cluster running on bare-metal hardware. I architected a hybrid cloud network using VMware NSX-V and Arista Hardware VTEPs (VXLAN Tunnel Endpoints), extending Layer 2 domains across clouds without modifying a single IP address.

The Challenge

The client’s application suite relied on hardcoded IP addresses embedded across thousands of configuration files.

Physical DB Constraint: Bare-metal Oracle RAC clusters could not be virtualized due to hardware license locks.
Layer 2 Extension Needs: Cloud-hosted application VMs required direct Layer 2 adjacency with physical database nodes.
Sub-Millisecond Latency SLA: Database transactions required low-latency, line-rate throughput across the hybrid boundary.

[!IMPORTANT] Extending Layer 2 subnets across WAN connections often creates routing loops and broadcast storms. The solution had to isolate broadcast domains while preserving hardware forwarding speed.

The Solution

We deployed an NSX-V SDDC (Software-Defined Data Center) on IBM Cloud Bare Metal Hosts linked to physical on-premise datacenter switches.

Technology Stack

Overlay Network: VMware NSX-V 6.4 (VXLAN Encapsulation)
Underlay Switches: Arista 7050SX (Hardware VTEP Binding)
Dedicated Transit: IBM Cloud Direct Link (10 Gbps Dedicated Fiber)
Routing Protocols: eBGP (Underlay) + OSPF (Overlay)

Technical Architecture

Physical Oracle RAC servers cannot run the NSX-V kernel module. To bridge them into the virtual overlay, we brought the VXLAN VTEP directly to the physical Top-of-Rack (ToR) Arista switches.

Arista Hardware VTEP: The Arista switch encapsulated raw Ethernet frames from physical Oracle DB servers into VXLAN packets (VNI 5001).
NSX-V Logical Switch: Cloud-based application VMs connected directly to VNI 5001 on ESXi hosts.
Transparent L2 Bridge: To the application tier, the physical Oracle DB appeared local to the same virtual switch.

[!NOTE] Hardware Offload: Running VXLAN encapsulation in Arista ASIC hardware maintained 10 Gbps line-rate throughput with sub-millisecond packet processing times.

Hybrid Overlay Architecture

graph TD
    subgraph "NSX-V Overlay Domain (Logical L2 - VNI 5001)"
    App[Cloud Virtual App Server] <-->|VXLAN Encapsulated Frame| DB[Physical On-Prem Oracle RAC]
    end

    subgraph "Physical Underlay Transport (Layer 3 BGP)"
    Host[ESXi Host Kernel VTEP] <-->|IBM Direct Link 10G| Spine[Arista Spine Switch]
    Spine <-->|BGP Underlay| ToR[Arista ToR HW-VTEP]
    end

    App -.-> Host
    DB -.-> ToR

Business Impact

Zero IP Re-Addressing: Successfully migrated 500+ enterprise workloads with 0% IP modification.
Performance Consistency: Delivered 10 Gbps line-rate throughput for high-volume database transactions.
Business Continuity: Completed datacenter evacuation 2 months ahead of lease expiration.

The Verdict

Key Takeaway

Decouple Logical Networks from Physical Underlays.

By combining VMware NSX-V with Arista Hardware VTEPs, you can stretch Layer 2 network boundaries across physical and cloud environments without incurring routing loop penalties.