The Ansible Tower Pipeline That Wiped the Wrong Environment

“The Ansible Tower job said ‘SUCCESS’. The NOC dashboard turned red 90 seconds later. Three production tenant segments had been deleted.”

The Setup

In late 2021, as a Systems Integration Advisor at NTT Data, we were running an Ansible Tower-based Zero-Touch Provisioning platform to automate NSX-T network lifecycle across 200+ vSphere host clusters.

The ZTP platform handled the complete stack: provisioning new Tier-1 gateways, deploying distributed firewall sections, configuring BGP peer relationships, and decommissioning tenant segments at contract end.

The teardown workflow had been running cleanly in staging for eight months. Fully idempotent, gracefully handled missing resources with ignore_errors: yes, and took about 4 minutes end-to-end.

In November, a new engineer joined the platform team. He had a legitimate task: run a teardown of three decommissioned development segments in the staging NSX-T cluster.

The Mess

The staging and production NSX-T Manager instances shared the same Ansible Tower credential vault. Both used the same playbook — nsxt_tenant_teardown.yml — with different survey variables passed at launch time.

The Tower survey prompt asked for two variables:

# Tower Job Template Survey Variables
vars:
  - variable: nsxt_manager_fqdn
    default: 'nsxt-staging.corp.internal'
    description: 'NSX-T Manager FQDN'

  - variable: tenant_segment_ids
    type: textarea
    description: 'Comma-separated Segment IDs to delete'

The new engineer launched the job. He typed the correct segment IDs. He did not change the nsxt_manager_fqdn default.

The default was nsxt-staging.corp.internal.

Except the DNS entry for nsxt-staging.corp.internal had been updated 48 hours earlier — the staging cluster had been rebuilt and given a new IP, and the DNS record had been temporarily left pointing to the old IP while the certificate was re-issued.

The old IP now belonged to nsxt-prod-secondary.corp.internal.

The NSX-T REST API authenticated successfully against production. The teardown ran cleanly.

# Ansible Tower Job Output (Job ID: 2347)
TASK [Delete Tenant Segment: VLAN-SEG-PROD-FIN-001] ****************************
2021-11-22 14:42:11: ok: [localhost] => (item=VLAN-SEG-PROD-FIN-001)

TASK [Delete Tenant Segment: VLAN-SEG-PROD-FIN-002] ****************************
2021-11-22 14:42:14: ok: [localhost] => (item=VLAN-SEG-PROD-FIN-002)

TASK [Delete Tenant Segment: VLAN-SEG-PROD-RETAIL-001] *************************
2021-11-22 14:42:18: ok: [localhost] => (item=VLAN-SEG-PROD-RETAIL-001)

PLAY RECAP *****************************
localhost                  : ok=7    changed=3    unreachable=0    failed=0

Job ID: 2347 | Status: SUCCESS

Three production finance and retail segments were deleted. The VMs attached to those segments lost all network connectivity. This included a payment gateway node that had processed ₹0 in revenue for the 8 minutes before the oncall team identified the root cause.

---

The Solution

Recovery first, controls second.

Step 1: Segment Recreation from NSX-T Backup

NSX-T Manager backed up automatically to an S3-compatible object store every 4 hours. The most recent backup was 3h 42m old.

Rather than restoring the full NSX-T configuration (which would have overwritten 4 hours of other config changes across 200 clusters), I extracted the three deleted segment configurations directly from the backup JSON:

# Extract deleted segment configs from NSX-T backup archive
tar -xzf nsxt-backup-2021-11-22T11:00Z.tgz nsxt-config.json

# Parse and extract specific deleted segments using jq
jq '.["nsxt-segment"][] | select(
  .id == "VLAN-SEG-PROD-FIN-001" or
  .id == "VLAN-SEG-PROD-FIN-002" or
  .id == "VLAN-SEG-PROD-RETAIL-001"
)' nsxt-config.json > deleted_segments.json

Then I used the NSX-T Policy API to recreate them in-place:

# POST each segment back to NSX-T Policy API (production)
while IFS= read -r segment; do
  SEGMENT_ID=$(echo "$segment" | jq -r '.id')

  curl -sk -u "admin:$NSX_PROD_PASSWORD" \
    -X PUT "https://nsxt-prod-primary.corp.internal/policy/api/v1/infra/segments/$SEGMENT_ID" \
    -H "Content-Type: application/json" \
    -d "$segment"

  echo "Restored segment: $SEGMENT_ID"
done < <(jq -c '.[]' deleted_segments.json)

Segments recreated in 6 minutes. VM network connectivity restored in 8 minutes (NSX-T overlay dataplane propagation). Total downtime: 16 minutes.

Step 2: Environment Isolation Rebuild

The root cause was a single-character assumption: that nsxt-staging.corp.internal always resolved to staging. We had no validation.

We rebuilt the ZTP platform with three mandatory controls.

Control 1: Environment Tag Verification Before Any Destructive Task

# playbooks/nsxt_tenant_teardown.yml (revised)
- name: SAFETY CHECK — Verify target NSX-T Manager is NOT production
  block:
    - name: Query NSX-T Manager environment tag
      uri:
        url: 'https://{{ nsxt_manager_fqdn }}/api/v1/node/properties'
        user: admin
        password: '{{ nsxt_password }}'
        validate_certs: false
      register: nsxt_properties

    - name: Fail immediately if environment tag is PRODUCTION
      fail:
        msg: 'ABORT: Target NSX-T Manager is tagged as PRODUCTION. Teardown cannot proceed. Use the correct staging FQDN.'
      when: "'PRODUCTION' in nsxt_properties.json.product_version | upper"
  rescue:
    - fail:
        msg: 'ABORT: Could not verify environment tag. Refusing to proceed with teardown.'

Control 2: Explicit Environment Confirmation Variable

# Tower Job Template Survey — Revised
vars:
  - variable: nsxt_manager_fqdn
    description: "NSX-T Manager FQDN (staging only)"

  - variable: environment_confirmation
    description: "Type STAGING to confirm this is not a production target"

- name: Confirm environment is STAGING before proceeding
  assert:
    that:
      - environment_confirmation | upper == "STAGING"
    fail_msg: "Environment confirmation failed. You must type STAGING to proceed."

Control 3: DNS Verification Against Known Production IP Block

# scripts/validate_nsxt_target.py
# Runs as pre-task before any teardown job in Tower

import socket
import ipaddress
import sys

PRODUCTION_CIDR = ipaddress.ip_network("10.200.0.0/16")
target_fqdn = sys.argv[1]

resolved_ip = socket.gethostbyname(target_fqdn)
if ipaddress.ip_address(resolved_ip) in PRODUCTION_CIDR:
    print(f"❌ ABORT: {target_fqdn} resolves to {resolved_ip} — within PRODUCTION CIDR {PRODUCTION_CIDR}")
    sys.exit(1)

print(f"✅ SAFE: {target_fqdn} → {resolved_ip} (not in production IP range)")
sys.exit(0)

Before / After Operational Metrics

Metric	Before Incident	After Controls
Environment Validation	None — assumed FQDN was correct	3-layer: NSX-T tag + user confirmation + DNS CIDR check
Teardown Pre-flight	No pre-flight checks	Mandatory verification playbook before any changed task
Recovery Time (segment restore)	Unknown (no tested procedure)	6 minutes (scripted from backup)

---

Key Takeaway

Idempotent automation is only half the safety equation.

A playbook that runs cleanly every time is not “safe” — it is reliable. Reliability without environment isolation is just a faster way to do the wrong thing.

Before any automation platform runs destructive operations (teardown, decommission, bulk delete), you need at least two out-of-band signals confirming the target is what the operator claims it is. FQDN resolution is not one of those signals — DNS TTLs and stale records mean it has a non-zero failure rate at the worst possible moment.

“Did it succeed?” is the wrong question. The right question is: “Did it succeed against the intended target?”

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin