Zero-Touch Datacenter Provisioning: Ansible Tower Meets NSX-T Policy API

“Manual network provisioning in a private cloud isn’t just slow — it is a silent guarantee of configuration drift across clusters.”

The Setup

In 2021, scaling private cloud infrastructure for financial enterprise clients meant managing over 200 vSphere host clusters across dual data centers. Every new tenant onboarded required dedicated network segments, distributed firewall policies, and Tier-1 gateway routing definitions.

On paper, we had modern VMware NSX-T software-defined networking installed. In practice, engineers were still logging into vCenter and NSX Manager manually to click through 40 UI screens per tenant request.

The workload was overwhelming. Network engineers spent 70% of their week copying VLAN IDs, typing CIDR blocks into IPAM, and hand-crafting firewall rules across multiple management consoles.

---

The Mess

The manual process was breaking down under high request volumes. A typical tenant onboarding ticket took 18 to 21 days from request submission to final sign-off.

The first attempt to solve this was a series of quick python scripts written by individual team members. This made things worse:

• Scripts were stored on local laptops without version control or rollback capabilities.
• Network engineers ran scripts with different parameter assumptions, creating duplicate Tier-1 Gateways.
• An unhandled API error during a Friday evening change window left 40 host clusters with half-applied Distributed Firewall Rules, dropping inter-VLAN routing for 4 hours.

[ERROR] 2021-09-10 18:42:11 - NSX-T REST API 500 Internal Error
POST https://nsx-mgr-01.corp.local/api/v1/firewall/sections
Payload: {"display_name": "Tenant-Alpha-DFW", "rules": [...]}
Error: Section ID conflict. Resource locked by concurrent session admin_sachin.

The realization hit during a late-night incident call: we didn’t need faster local scripts. We needed an idempotent, centralized Zero-Touch Provisioning (ZTP) pipeline with audit logging and RBAC.

---

The Solution

I designed a centralized Ansible Tower (AWX) workflow that treated network infrastructure as declarative YAML definitions. Instead of imperative step-by-step CLI commands, engineers committed tenant specs directly into Git.

Here is the exact declarative YAML structure we established for tenant definitions:

# host_vars/tenant_alpha.yml - Declarative NSX-T Tenant Definition
tenant_id: 'tenant-alpha'
tier1_gateway: 't1-gw-alpha'
edge_cluster_id: '7a8b9c0d-1234-5678-90ab-cdef12345678'

subnets:
  - name: 'web-subnet'
    cidr: '10.240.10.0/24'
    gateway: '10.240.10.1/24'
  - name: 'app-subnet'
    cidr: '10.240.20.0/24'
    gateway: '10.240.20.1/24'

firewall_rules:
  - name: 'allow-web-to-app'
    source: ['web-subnet']
    destination: ['app-subnet']
    service: 'HTTPS'
    action: 'ALLOW'

The Ansible Tower job template consumed the declarative spec and executed idempotent REST calls to the NSX-T Policy API:

# roles/nsxt_tenant/tasks/main.yml - Idempotent NSX-T Policy API Pipeline
- name: Create Tier-1 Gateway via NSX-T Policy API
  uri:
    url: 'https://{{ nsxt_manager }}/policy/api/v1/infra/tier-1s/{{ tenant_id }}'
    method: PUT
    url_username: '{{ nsxt_user }}'
    url_password: '{{ nsxt_password }}'
    validate_certs: false
    body_format: json
    body:
      id: '{{ tenant_id }}'
      display_name: '{{ tenant_id }}'
      tier0_path: '/infra/tier-0s/t0-core-gateway'
      route_advertisement_types:
        - 'TIER1_CONNECTED'
        - 'TIER1_STATIC_ROUTES'
    status_code: [200, 201]
  register: nsxt_tier1_result

- name: Create Segment Subnets under Tier-1 Gateway
  uri:
    url: 'https://{{ nsxt_manager }}/policy/api/v1/infra/tier-1s/{{ tenant_id }}/segments/{{ item.name }}'
    method: PUT
    url_username: '{{ nsxt_user }}'
    url_password: '{{ nsxt_password }}'
    validate_certs: false
    body_format: json
    body:
      id: '{{ item.name }}'
      display_name: '{{ item.name }}'
      subnets:
        - gateway_address: '{{ item.gateway }}'
      transport_zone_path: '/infra/sites/default/enforcement-points/default/transport-zones/tz-overlay'
    status_code: [200, 201]
  loop: '{{ subnets }}'

The Results

The automated Ansible Tower pipeline transformed private cloud agility across our 200-cluster estate:

• Provisioning Time: Reduced from 21 days to 90 seconds.
• Configuration Errors: Dropped from 35% manual error rate to 0% schema failure.
• Audit Trail: 100% of network changes tracked via Git commit history and Ansible Tower execution logs.

---

Key Takeaway

If your automation pipeline requires manual CLI overrides or UI clicks, it is not automation — it is a script. True NetDevOps relies on declarative state and idempotent API enforcement.

---

Related Reading:

• The Ansible Tower Pipeline That Wiped the Wrong Environment — What happens next when this same platform runs teardown against the wrong cluster
• The Kubernetes CNI Bug That Isolated 140 Production Pods — The NCP CNI incident this ZTP platform fed into

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin