Recovering from a 400-Resource Terraform State Drift Emergency

“The scariest output in infrastructure engineering is ‘Plan: 412 to add, 0 to change, 412 to destroy’ against a live production state.”

The Setup

In 2022 at NTT Data, our private cloud management plane was managed via Terraform vSphere provider modules. The .tfstate file tracked over 400 active virtual machines, virtual switches, distributed port groups, and storage DRS clusters across multiple datacenters.

The gold standard rule of Infrastructure-as-Code (IaC) is strict: all infrastructure modifications must go through Terraform. No manual vCenter console tweaks allowed.

When teams honor this contract, infrastructure state remains predictable, reproducible, and easily audit-logged.

---

The Mess

During a high-priority incident on a Thursday evening, a senior sysadmin bypassed the Git pipeline to manually re-assign 80 production virtual machines to different port groups and vSphere datastores directly inside the vCenter UI.

No one updated the underlying HCL code or state file. On Friday morning right before a major client SLA review, a scheduled terraform plan was triggered:

• Terraform read the out-of-band vCenter changes and concluded the state was completely out of sync.
• The execution plan indicated Terraform would destroy and recreate 412 production VMs to restore the state defined in Git.
• To make matters worse, an interrupted CLI execution left the remote DynamoDB State Lock stuck in a locked state.

[ERROR] 2022-05-27 08:14:02 UTC - Terraform Plan Output
Error: Error acquiring the state lock: ConditionalCheckFailedException:
Lock Info:
  ID:        c7d2e90f-482a-4a1b-9012-3456789abcde
  Path:      tf-state-bucket/prod/vsphere.tfstate
  Operation: OperationTypePlan
  Who:       admin_sachin@ops-workstation-01
  Created:   2022-05-27 08:10:00 UTC

Plan: 412 to add, 0 to change, 412 to destroy.

Running terraform apply meant catastrophic outage for 400 live tenant workloads. Panic spread across the NOC.

---

The Solution

I immediately halted all pipeline executions and began surgical state reconciliation. The objective: align Terraform .tfstate with actual vCenter reality without triggering resource recreation.

First, we force-unlocked the DynamoDB state lock:

# Force unlock stuck remote state lock ID
terraform force-unlock -force "c7d2e90f-482a-4a1b-9012-3456789abcde"

Next, rather than letting Terraform destroy live VMs, I updated the HCL code to match the new vCenter parameters and selectively refreshed target resources:

# Targeted state refresh to pull live vCenter attributes without applying changes
terraform refresh -target=module.compute_cluster.vsphere_virtual_machine.app_nodes[0]

For resources with severe schema mismatches, we selectively removed them from state and re-imported their live vCenter UUIDs:

# Remove corrupted state references without deleting physical VMs
terraform state rm module.compute_cluster.vsphere_virtual_machine.app_nodes[12]

# Re-import live vCenter VM UUID into Terraform state
terraform import module.compute_cluster.vsphere_virtual_machine.app_nodes[12] "/Datacenter/vm/Prod-App-Node-12"

The Results

Through systematic state manipulation and HCL alignment, we fully restored state integrity:

• Production Outage: 0 seconds downtime across all 412 tenant VMs.
• State Reconciliation: Reconciled 400+ drift anomalies in 45 minutes.
• Policy Update: Revoked direct write permissions in vCenter, enforcing 100% PR-only Terraform execution.

---

Key Takeaway

State drift is inevitable when humans have direct console access. Never run terraform apply when faced with massive resource destruction — use targeted terraform refresh, state rm, and import to reconcile state safely.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin