We Built a Tier-3 Data Center From Scratch. Here Is What Nobody Tells You.

“Anyone who says they built a Tier-3 data center the right way has never actually built one.”

I built two of them. Not consulted on, not reviewed — built. From empty floor space to live production environments taking real customer traffic.

The first time, I had no idea what I was doing. The second time, I had scars from the first.

---

The Setup

Net4 India handed me the second DC project in 2012. The first one was mostly done by then, running about 120 production servers across three racks. That one taught me things that no Tier-3 certification course covers.

The goal for DC2 was bigger: 400 planned server slots, dual-path power (2N), N+1 cooling, and a full dual-stack IPv4/IPv6 BGP topology. We were an ISP. We needed to be able to announce our own IP space from two geographically separated sites.

The budget was approximately ₹0 for consultants. We had Cisco gear, a trusted electrician, and a team of three.

---

The Mess

Cooling First. Not Networking.

The standard career path in networking teaches you to think in packets. That instinct will kill you in a data center build.

The first mistake I made in DC1 was racking servers before the PAC units were load-tested. You get the racks up, the switches are happy, the servers are pinging — and then three weeks later, at 2am on a June night in Delhi, the PAC unit trips because the thermal load crossed the design threshold.

It was 46°C outside. The DC was on the third floor of a building with no backup HVAC.

We lost six servers that night to thermal shutdown. Not permanently — but the recovery took four hours, and the postmortem cost us two weeks of arguing with the real estate vendor about cold aisle containment.

The lesson: rack utilization density must be planned before anything else is powered on. A half-full rack runs cooler than a full one. Obvious in retrospect. Nobody told me.

The P2V Migration That Almost Took Out Oracle

By 2013 we were migrating legacy physical servers to VMware ESXi. Standard P2V using VMware Converter — take a snapshot of the running machine, push it to a VM, test, cut over.

Except one server was running Oracle 11g on RHEL 5. The DBA had the licensing tied to the physical MAC address. The Converter run completed fine. The VM booted fine. Oracle refused to start.

First diagnosis: bad export. We ran the Converter twice. Same result.

Second diagnosis: initrd issue. We spent a full day rebuilding the initrd on the VM to match the physical hardware driver set. Oracle still refused.

Third diagnosis: actual cause. The Oracle license manager was checking both the MAC address and the CPU count. The physical box was a dual-socket Xeon. The VM had been created with 8 vCPUs spread across 2 virtual sockets. Oracle saw a different socket topology and treated it as a different machine.

The fix was two lines in the .vmx file:

# Force Oracle to see the same physical socket topology as the source server
numvcpus = "8"
cpuid.coresPerSocket = "4"

We didn’t know VMware let you tune socket topology at that level. The vendor didn’t tell us. We found it in a VMTN forum post from 2009.

Total time lost: 11 days. Oracle support ticket: still open at the time we cut over.

---

The Solution

MP-BGP Dual-Stack — Because Somebody Had to Do It

This is the part I’m genuinely proud of.

In 2013, IPv6 was theoretical at most Indian ISPs. We had the address space (a /32 from APNIC), we had two upstream providers who could announce it, and we had a team that had never operated a dual-stack BGP network before.

The topology we built:

                  [Upstream 1 — TATA]          [Upstream 2 — Airtel]
                        |                              |
                        +---------- iBGP Peering ------+
                                        |
                              [Core Router — Cisco 7206]
                              BGP AS: 9829
                              Prefixes: 203.xx.xx.0/24 (IPv4)
                                        2405:xxxx::/32 (IPv6)
                                        |
                          +-------------+-------------+
                          |                           |
                    [DC1 Edge]                  [DC2 Edge]
                    (Noida Site A)             (Noida Site B)

We ran MP-BGP with dual address families: address-family ipv4 unicast and address-family ipv6 unicast on the same iBGP sessions. The upstream peers spoke both.

The real engineering challenge was route policy. We needed to prefer different upstreams for different traffic types — IPv4 via TATA (lower latency), IPv6 via Airtel (better peering). We did it with local-preference manipulation on inbound route maps:

# Prefer TATA for IPv4 (higher local-pref = preferred)
route-map SET_LP_PREFER_TATA permit 10
 match ip address prefix-list OUR_IPV4_SPACE
 set local-preference 200

# Prefer Airtel for IPv6
route-map SET_LP_PREFER_AIRTEL permit 10
 match ipv6 address prefix-list OUR_IPV6_SPACE
 set local-preference 200

It worked. IPv6 traffic took a different path than IPv4. We verified it with traceroute6 and with Cacti’s Spine C-poller showing separate graphs for the two address families.

Cacti Spine — Why C Poller Matters at Scale

By 500 monitored devices, the standard PHP poller in Cacti starts dropping polls. You miss 15-second intervals, you get gaps in graphs, and on-call engineers start getting paged for “outages” that are actually graph gaps.

We switched to spine — the compiled C-based poller — after the first data center went live:

# /etc/cactid.conf
DB_Host        localhost
DB_Database    cacti
DB_User        cactiuser
DB_Pass        xxxxxx
Threads        20
# 20 parallel threads, each handling a partition of the device list

Twenty threads. Previously the PHP poller was running single-threaded. Graph gap rate dropped from ~12% to <0.3% within a week.

Small config change. Enormous operational difference.

---

The Verdict

Key Takeaway

A data center is not a network. It is a building with a network inside it.

If you approach a DC build thinking in protocols and switching, you will get the cooling wrong. The physics comes before the packets. Power density, hot aisle/cold aisle separation, and PAC unit capacity planning are not “facilities team problems” — they are your problems if you are the person responsible for keeping the servers alive.

The P2V/Oracle incident is a reminder that vendor tools lie by omission. VMware Converter will happily migrate a machine and not tell you that the socket topology change will break your licensing. You have to know what questions to ask before the migration, not after the rollback.

And the MP-BGP dual-stack? That ran in production for three years, serving real customer traffic, before anyone noticed we’d quietly shipped native IPv6. Most peers were still “planning” their IPv6 strategy. We were routing packets.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin