The Kubernetes CNI Bug That Isolated 140 Production Pods at Midnight

“The Kubernetes API reported that all 140 microservices were running healthy. Every single HTTP probe from the ingress gateway was failing with a 504 Gateway Timeout.”

The Setup

In 2022, as a Systems Integration Advisor scaling private clouds, I architected the integration between enterprise Kubernetes worker clusters and VMware NSX-T Data Center using the NSX Container Plugin (NCP) CNI.

NCP translates Kubernetes native objects — Namespaces, Services, NetworkPolicies, and Pods — directly into NSX-T Policy API primitives. A Kubernetes NetworkPolicy becomes an NSX Distributed Firewall (DFW) section. A Kubernetes Service becomes an NSX L4 load balancer virtual server.

The promise was seamless zero-trust microsegmentation. Developers wrote standard apiVersion: networking.k8s.io/v1 YAML, and the infrastructure team got hardware-accelerated ESXi hypervisor firewall enforcement.

The Mess

At 11:42 PM on a Sunday night, the platform team deployed a rolling upgrade to the payment gateway namespace. Standard CI/CD pipeline push.

Within three minutes, 140 production pods across three worker nodes lost all cross-namespace communication.

The Kubernetes control plane showed all pods in Running 1/1 status. Readiness probes passing on localhost. But API calls between the checkout service and the inventory service timed out after 30 seconds.

# Log snippet from checkout-service pod stdout
2022-11-13 23:45:12.408 ERROR [http-nio-8080-exec-4] c.e.payment.client.InventoryClient:
  Connection timed out: api.inventory.prod.svc.cluster.local/10.244.12.88:8080
java.net.SocketTimeoutException: connect timed out
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)

The first diagnosis was an overlay MTU mismatch. We checked the GENEVE tunnel MTU — set to 1600 bytes, running clean with zero drop counters on the ESXi host VTEPs.

Then we checked the NCP CNI controller logs on the k8s master node:

# NCP CNI Controller Log Trace (ncp-container-plugin-5f7b8d-x9q4)
2022-11-13T23:45:01.882Z ERROR ncp.nsx.dfw_builder Failed to translate NetworkPolicy default/allow-payment-to-inventory
  nsx_api_error: 500 Internal Server Error
  response_body: {"error_code": 4012, "error_message": "DFW Section limit exceeded or invalid Rule syntax: target IPSet contains overlapping CIDR block"}
2022-11-13T23:45:02.104Z WARN ncp.nsx.sync Retrying NSX Policy API sync (attempt 5/5). Stale DFW section left in UNENFORCED state.

The bug was subtle. When developers updated the namespace selector labels, NCP attempted to update the corresponding NSX DFW section via the NSX Policy API.

Due to a race condition in NCP version 3.2.1 when handling simultaneous namespace label updates and pod IP recycling, NCP created duplicate IPSet objects in NSX-T. The NSX Policy API returned a HTTP 500 Internal Error, but NCP’s local cache assumed the sync succeeded.

The result: NSX DFW applied a default DENY ALL catch-all rule while dropping the newly translated ALLOW rules. All ingress traffic to 140 pods was silently dropped at the hypervisor kernel level.

---

The Solution

We executed emergency remediation in two phases — immediate outage recovery, followed by an automated NetDevOps validation pipeline.

Phase 1: Emergency DFW Section Flush

I wrote an emergency Ansible playbook to query NSX-T Policy API directly, detect orphaned DFW sections generated by NCP, and force-reconcile the state.

# playbooks/nsxt_ncp_reconcile.yml
# Direct NSX-T Policy API reconciliation for NCP CNI state drift
---
- name: Reconcile NSX-T DFW Rules for Kubernetes CNI
  hosts: localhost
  gather_facts: false
  vars:
    nsx_mgr: '172.16.10.10'
    domain_id: 'default'

  tasks:
    - name: Fetch all NCP-managed DFW Security Policies
      amazon.aws.uri:
        url: 'https://{{ nsx_mgr }}/policy/api/v1/infra/domains/{{ domain_id }}/security-policies'
        method: GET
        user: 'admin'
        password: '{{ nsx_password }}'
        validate_certs: false
      register: dfw_policies

    - name: Identify and Purge Stale Orphanted DFW Sections
      amazon.aws.uri:
        url: 'https://{{ nsx_mgr }}/policy/api/v1/infra/domains/{{ domain_id }}/security-policies/{{ item.id }}'
        method: DELETE
        user: 'admin'
        password: '{{ nsx_password }}'
        validate_certs: false
      loop: '{{ dfw_policies.json.results }}'
      when:
        - item.marked_for_delete is defined and item.marked_for_delete == true
        - "'k8s-ncp' in item.display_name"

    - name: Trigger NCP Pod Restart to Force Full CNI Resync
      kubernetes.core.k8s:
        state: restarted
        kind: Deployment
        name: nsx-ncp
        namespace: nsx-system

Running the playbook flushed the stale NSX DFW sections. NCP restarted, queried the Kubernetes API master for the current cluster state, and cleanly reconstructed 42 valid DFW firewall sections in 45 seconds.

Traffic restored. Total outage duration: 38 minutes.

Phase 2: Automated NetDevOps Validation Pipeline

To prevent future CNI translation drift, we built a Python validation daemon (ncp_cni_auditor.py) that periodically compares the Kubernetes NetworkPolicy specification against the active NSX DFW ruleset.

# scripts/ncp_cni_auditor.py
# NetDevOps CNI State Verification Daemon

import requests
from kubernetes import client, config

def audit_cni_policy_sync():
    config.load_incluster_config()
    k8s_api = client.NetworkingV1api()

    # Get all active Kubernetes NetworkPolicies
    k8s_policies = k8s_api.list_network_policy_for_all_namespaces()
    k8s_policy_names = {p.metadata.name for p in k8s_policies.items}

    # Query NSX-T Policy API for active DFW policies
    nsx_url = "https://172.16.10.10/policy/api/v1/infra/domains/default/security-policies"
    resp = requests.get(nsx_url, auth=("admin", "NSX_SECRET"), verify=False)
    nsx_policies = resp.json().get("results", [])

    nsx_policy_names = {
        p["display_name"].replace("k8s-policy-", "")
        for p in nsx_policies
        if p.get("display_name", "").startswith("k8s-policy-")
    }

    # Detect drift: policies present in K8s but missing in NSX-T DFW
    drifted = k8s_policy_names - nsx_policy_names
    if drifted:
        print(f"❌ CNI DRIFT DETECTED! Unenforced policies: {drifted}")
        trigger_pagerduty_alert(drifted)
    else:
        print("✅ NCP CNI & NSX-T DFW fully in sync.")

if __name__ == "__main__":
    audit_cni_policy_sync()

Before / After Operational Metrics

Metric	Before Fix	After Fix
CNI Policy Sync Detection Time	72 hours (manual report)	< 30 seconds (automated daemon)
Orphaned DFW Section Recovery	2.5 hours manual API calls	45 seconds automated playbook
Cross-Namespace Microsegmentation SLA	98.2%	99.99%

---

Key Takeaway

Never assume a Kubernetes CNI plugin is a transparent pipe. When abstracting low-level firewall or routing hardware behind high-level Kubernetes YAML, the translation layer is your primary point of failure.

If you run overlay CNIs (NSX NCP, Cilium, Calico) in enterprise production, monitor the translation layer itself — not just Kubernetes pod status or host ping. Query the underlying firewall hardware API directly and alert on state drift before an unapplied policy isolates your microservices.

---

Related Reading:

• Zero-Touch Datacenter Provisioning: Ansible Tower Meets NSX-T Policy API — The ZTP platform this incident lived on
• The Ansible Tower Pipeline That Wiped the Wrong Environment — What happens when automation runs clean against the wrong target
• Translating Kubernetes NetworkPolicies to NSX-T DFW Rules — The architecture context behind the NCP CNI design

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin