Bridging Physical Bare-Metal to VMware NSX Overlays with Arista HW-VTEP & OVSDB

“Virtualizing networks is easy until you hit physical bare-metal database servers that cannot run a hypervisor VTEP.”

The Setup

In March 2019 at IBM, we architected a hybrid software-defined data center (SDDC) solution for a enterprise telecom client. The application tier ran inside VMware NSX virtualized overlay networks (VXLAN / GENEVE), but the backend high-transaction database tier ran on physical Oracle RAC bare-metal servers connected directly to Arista 7050SX Leaf switches.

To enforce Zero-Trust micro-segmentation, physical Oracle RAC servers needed to communicate directly with virtualized application workloads over the same Layer 2 logical segment without passing through external firewalls or routers.

The Mess

Software VTEPs (Virtual Tunnel Endpoints) run inside ESXi hypervisors, but bare-metal physical servers have no hypervisor.

Our initial attempt to use software routing gateways caused packet latency to jump from 0.8ms to 28ms, degrading Oracle RAC inter-node cache synchronization. During peak load, OVSDB control plane state dropped:

[CRITICAL] 2019-03-14 11:45:00 UTC - Arista EOS Log (Switch: Leaf-01A)
%OVSDB-3-CONTROLLER_DISCONNECTED: OVSDB controller 10.200.1.15 (NSX Controller) disconnected.
%OVSDB-4-VTEP_MAC_TABLE_FULL: Hardware VTEP MAC Table capacity exceeded on VTEP interface Vxlan1.
Hardware VTEP Status: Physical Switch Leaf-01A failed to program 1,200 MAC addresses.
Application Result: Bare-metal Oracle DB connection dropped. Transaction processing halted.

sequenceDiagram
    autonumber
    participant OracleDB as Bare-Metal Oracle DB (Physical)
    participant Arista as Arista 7050SX (HW-VTEP)
    participant NSXCtrl as NSX Controller (OVSDB)
    participant ESXi as ESXi Host (GENEVE VM)

    OracleDB->>Arista: L2 Frame (VLAN 100)
    Arista->>NSXCtrl: OVSDB Query: Lookup MAC -> VNI 5001
    NSXCtrl-->>Arista: VTEP Tunnel Endpoint: 10.200.1.50
    Arista->>ESXi: GENEVE Encapsulated Packet (VNI 5001)
    ESXi-->>OracleDB: Decapsulated Frame to App VM

The Solution

I architected a hardware-accelerated Arista Hardware VTEP & OVSDB Integration with NSX Controllers:

1. Hardware VTEP Handoff: Bound Arista 7050SX physical switch ports to NSX Logical Switches via OVSDB (Open vSwitch Database Management Protocol).
2. Head-End Replication (HER): Replaced multicast BUM replication with Head-End Unicast Replication, offloading packet replication to Arista hardware ASICs.
3. MAC Limit Tuning & Pruning: Enforced strict MAC limits per switch port and pruned unused VLAN-to-VNI mappings.

! Arista EOS Configuration - Hardware VTEP & OVSDB Integration
cvx
  no shutdown
  service ovsdb
    no shutdown
!
interface Vxlan1
  vxlan source-interface Loopback1
  vxlan udp-port 4789
  vxlan controller-client nsx-controller-01 10.200.1.15
  vxlan flood vlan 100,101 head-end replication

# Verify OVSDB Hardware VTEP Connection State on Arista Switch
Leaf-01A# show ovsdb controller
Controller: 10.200.1.15:6640
  Status: CONNECTED (SSL)
  Role: Master
  VTEP Name: Leaf-01A-VTEP
  Logical Switches Programmed: 12
  Hardware MAC Table Entries: 450 / 8192 (OPTIMAL)

The Results

The Arista Hardware VTEP deployment seamlessly bridged physical bare-metal servers into NSX overlays:

• Latency Overhead: Reduced bridging packet latency from 28ms (software gateway) to 0.6ms (Arista ASIC line rate).
• Throughput: Supported 40 Gbps hardware line-rate traffic between physical Oracle RAC nodes and virtual app VMs.
• Control Plane Stability: 100% OVSDB connection stability with 0 MAC table drops.

Key Takeaway

When connecting physical bare-metal servers into VMware NSX overlays, deploy Arista Hardware VTEPs with OVSDB and Head-End Unicast Replication (HER) to process encapsulation directly in switch hardware ASICs at line rate.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin