“The application team’s response to our re-IP plan was six minutes of silence on a conference call, followed by: ‘Do you know how many connection strings we have?’ We checked. It was 847. The re-IP plan died there.”

This is a story about a manufacturing client evacuating a physical datacenter into IBM Cloud (SoftLayer, as it was still called then). 500 production VMs. A hard deadline. And an Oracle RAC cluster that had not been touched since 2011 because nobody wanted to be the engineer who broke payroll.

The ask was straightforward. The constraint was not.

---

The First Plan (And Why It Failed in the First Meeting)

Every cloud migration playbook says the same thing: plan for re-IP.

The logic is clean on paper. You’re moving to a new network. New subnet allocations. Clean topology. The IP addressing you inherited from 2008 can finally die.

What the playbooks don’t say is what happens when you open that first application dependency map and find:

• 847 hardcoded database connection strings across 12 application teams
• A Checkpoint R77 firewall ruleset with 2,300 rules tied to specific /24 boundaries
• An Oracle RAC cluster whose listener IPs were baked into a vendor support contract — meaning changing them required a service request, a review, and a 6-week SLA

I had this conversation with the client’s application lead on a Tuesday afternoon in November 2018. He was polite about it. He said, “Sachin, I appreciate the technical correctness of your proposal. We are not doing that.”

He was right.

---

The Mess: What We Tried Before the Real Solution

The first instinct was to script around it.

I spent two days building a Python inventory tool using netmiko to extract every firewall rule referencing any of the migration subnets. The idea was to generate a “re-IP delta” — a mapping of old IPs to new IPs — and apply it systematically.

The tool worked. The delta had 4,400 entries.

The change management team looked at 4,400 entries across 12 application teams and asked how long the testing window would be. I said “three months minimum.” They said the project deadline was six weeks.

We needed a different approach.

The second idea was to use NAT translation at the IBM Cloud edge — keep the VMs on their original IPs, NAT at the boundary. That conversation lasted about 15 minutes before the Oracle RAC team pointed out that RAC inter-node communication would break the moment any packet hit a NAT boundary. Oracle licensing was involved. The meeting ended.

It was now week two. We had ruled out the two most common approaches. The project manager had started CC’ing the CTO on emails.

---

The Actual Solution: Stretching the L2 Domain Across a WAN

The answer was already in the VMware HCX documentation. I had read past it twice because it seemed too good to be true.

Layer-2 Network Extension.

The concept: instead of moving VMs to a new network, bring the network with them. Extend the on-premise VLAN — broadcast domain and all — directly into the IBM Cloud SDDC over a dedicated WAN link. The VMs never know they moved. Their IP addresses, default gateways, and ARP tables stay identical.

The architecture that made it work:

• VMware HCX WAN Interconnect: A pair of HCX appliances (on-premise and cloud-side) create an encrypted, deduplicated tunnel over the 10Gbps IBM Direct Link. This is the transport layer.
• HCX Layer-2 Extension: Specific VLANs are stretched across the tunnel. On-premise VLAN 102 appears as an NSX-T segment inside the IBM Cloud SDDC. Same subnet. Same gateway. No re-IP.
• Arista HW-VTEP Bridge: The on-premise physical switches (Arista 7050s) participate in the VXLAN fabric as hardware VTEPs via OVSDB protocol. This was the critical piece — it meant the physical servers (the Oracle RAC nodes) were also part of the stretched segment, without needing to be virtualised.
• HCX Bulk Migration: VMs replicate asynchronously to the cloud side while still running on-premise. WAN deduplication means only changed blocks travel the wire after the initial sync. When delta reaches near-zero, switchover takes approximately 2 seconds.

# Verifying HCX L2 extension tunnel health across the Direct Link
hcx-cli network-extension status --site-pair ONPREM_TO_IBM_CLOUD
# Expected output: Tunnel state: CONNECTED | Stretched VLANs: 102, 108, 115, 201

# Script we used to validate MAC table consistency across the stretched segment
# Run on both sides before each migration wave to detect any stale ARP entries
import paramiko

def check_mac_table(host, vlan_id):
    # SSH into the on-premise Arista and pull the MAC table for the stretched VLAN
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(host, username='netadmin', key_filename='/etc/ssh/migration_key')
    _, stdout, _ = client.exec_command(f'show mac address-table vlan {vlan_id}')
    return stdout.read().decode()

Migration Wave Planning

We did not migrate all 500 VMs at once. That would have been optimistic to the point of recklessness.

We ran four waves over three weekends. Each wave was preceded by 72 hours of asynchronous replication, a MAC table validation pass, and a signed-off rollback procedure that involved exactly one command: telling HCX to re-anchor the VMs back to the on-premise side.

Wave order mattered. We migrated stateless web-tier VMs first, then application servers, then database replicas, and left the Oracle RAC primary nodes for last — running them across the stretched segment in place rather than migrating them at all, until the vendor could schedule a proper maintenance window.

The payroll system ran through all four migration weekends without a single alert.

---

What This Tells You About Cloud Migration Strategy

The standard advice — plan for re-IP — is not wrong. It is the right answer for greenfield migrations or new applications.

It is the wrong answer when your application layer is older than your youngest team member.

Network topology and physical location are separate concerns. Most migration projects treat them as the same problem, which is why most migration projects run 6 months late and cost twice the estimate.

Extend the L2 domain first. Migrate the compute. Clean up the topology from inside the destination, on your schedule, after the deadline pressure is gone.

The re-IP happened eventually. Six months after cutover, the application teams had done the connection string updates at their own pace. The Checkpoint rules were replaced by NSX-T DFW policies. The Oracle RAC listeners were updated during a planned maintenance window.

Nobody missed a deadline. Nobody broke payroll.

Key Takeaway

Decouple Network Topology from Physical Location. Before you put re-IP on the migration critical path, count your hardcoded connection strings. If the number is over 100, you need Layer-2 Extension or you need more time than the business will give you.

HCX + HW-VTEP is not a permanent architecture. It is a bridge — a tool that buys you the time to do the topology cleanup right, without holding the migration hostage to application team delivery schedules.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin