The Azure Management Group Hierarchy That Broke 14 Business Units at Once

“We applied a single Azure Policy at the root Management Group level on a Friday afternoon. By Monday morning, 14 separate business units had filed P1 incidents.”

The Setup

In early 2023, in my current role, I was brought in as cloud architecture lead to design and implement an Azure Enterprise Scale Landing Zone for a global enterprise client consolidating 14 business units onto a single Azure tenant.

The client had been running Azure in what I can only describe as freestyle mode. Every BU had its own EA enrollment administrator, its own subscription naming convention, and its own firewall rule philosophy. Two BUs were running Azure Firewall Premium. One was running NVAs from a vendor I had never heard of.

The brief was standard CAF work: establish a Management Group hierarchy, impose Policy-as-Code guardrails, implement ExpressRoute hub-and-spoke topology, and onboard all 14 BUs without breaking their existing workloads.

The client’s CISO was watching this closely. The CTO had already promised the board a “secure, governed cloud footprint” in Q1.

The Mess

We spent three weeks designing the Management Group hierarchy using the CAF enterprise-scale reference architecture. Terraform modules, azurerm_management_group resources, policy assignment HCL — clean and reproducible.

On a Thursday afternoon, we applied a blanket Deny Azure Policy to restrict VM SKUs to approved series only. The assignment target was MG-Corp, the root of the corporate hierarchy.

# terraform/policy_assignments.tf
# Restrict allowed VM SKUs to approved production series only
resource "azurerm_management_group_policy_assignment" "restrict_vm_skus" {
  name                 = "restrict-vm-skus-corp"
  management_group_id  = azurerm_management_group.corp.id
  policy_definition_id = data.azurerm_policy_definition.allowed_skus.id
  enforce              = true

  parameters = jsonencode({
    listOfAllowedSKUs = {
      value = ["Standard_D4s_v5", "Standard_D8s_v5", "Standard_E8s_v5"]
    }
  })
}

What we did not notice — and what the Terraform plan output did not surface clearly — was that MG-Corp was the parent of MG-Sandbox and MG-Dev, where seven of the 14 BUs ran their CI/CD automation pipelines.

Those pipelines used Standard_B2s and Standard_D2s_v3 instances as build agents. Both were outside the approved SKU list.

By Friday noon, Terraform apply was complete. The policy inheritance propagated silently across the entire Management Group tree.

By Monday morning:

# Azure Activity Log — Error Pattern (filtered from 14 BU subscriptions)
2023-03-06T08:12:44Z ERROR | Resource: Microsoft.Compute/virtualMachines
  Status: RequestDisallowedByPolicy
  Policy Assignment: /providers/Microsoft.Management/managementGroups/MG-Corp/providers/Microsoft.Authorization/policyAssignments/restrict-vm-skus-corp
  Policy Definition: /providers/Microsoft.Authorization/policyDefinitions/cccc23c7-8427-4f31-ad1...
  Non-compliant Resource: Standard_B2s
  Description: The resource 'build-agent-ci-001' was disallowed by policy.

CI/CD pipelines across seven BUs had been failing silently all weekend. No alerts fired because the pipelines returned exit code 0 — they queued the VM deployment, the Azure ARM layer rejected it, and the pipeline runner reported “no resources to create.”

The next policy propagation check would only fire on the next terraform plan run — which was scheduled for Tuesday.

Meanwhile, dev teams across three time zones were staring at stuck deployments and attributing it to “Azure being weird.”

---

The Solution

Emergency fix was a four-step process.

Step 1: Policy Assignment Audit

I ran an immediate Azure Resource Graph query to identify every subscription inheriting the problematic assignment:

# Run Azure Resource Graph query to find all non-compliant resources
az graph query -q "
  PolicyResources
  | where type == 'microsoft.policyinsights/policystates'
  | where properties.complianceState == 'NonCompliant'
  | where properties.policyAssignmentId contains 'restrict-vm-skus-corp'
  | project subscriptionId, resourceGroup, resourceId, properties.resourceType
  | order by subscriptionId asc
" --query "data" -o table

The output listed 47 non-compliant resources across 11 subscriptions. All were CI/CD build agents.

Step 2: Emergency Policy Exemption

Rather than rolling back the policy (which would have created a security regression), we applied targeted Policy Exemptions scoped to the MG-Dev and MG-Sandbox Management Groups:

# terraform/policy_exemptions.tf
# Emergency exemption for CI/CD sandbox and dev MG tiers
resource "azurerm_management_group_policy_exemption" "dev_build_agents" {
  name                            = "dev-build-agent-sku-exemption"
  management_group_id             = azurerm_management_group.dev.id
  policy_assignment_id            = azurerm_management_group_policy_assignment.restrict_vm_skus.id
  exemption_category              = "Waiver"
  description                     = "Emergency waiver: CI/CD build agents require B-series SKUs. See incident INC-20230306."
  expires_on                      = "2023-09-08T23:59:59Z"  # 6-month review window
}

Exemption applied in 8 minutes. Policy propagation completed in 22 minutes. All 47 blocked resources resolved.

Step 3: CAF Hierarchy Redesign

The root cause was a structural design flaw: we had not separated Platform and Application Management Groups properly. The CAF reference architecture prescribes this clearly — but we had collapsed it under time pressure.

The correct hierarchy:

graph TD
    Root["MG-Root (Tenant Scope)"]
    Platform["MG-Platform (Identity/Hub/Shared)"]
    Corp["MG-Corp (Production Workloads)"]
    Dev["MG-Dev (Dev/Sandbox Tiers)"]

    HubNet["MG-Hub-Network"]
    Identity["MG-Identity"]
    Prod["MG-Prod Subscriptions"]
    CorpShared["MG-Corp-Shared"]
    DevBU1["MG-Dev-BU01"]
    DevBU2["MG-Dev-BU02"]
    Sandbox["MG-Sandbox (CI/CD Pipelines)"]

    Root --> Platform
    Root --> Corp
    Root --> Dev

    Platform --> HubNet
    Platform --> Identity

    Corp --> Prod
    Corp --> CorpShared

    Dev --> DevBU1
    Dev --> DevBU2
    Dev --> Sandbox

    style Root fill:#1e293b,stroke:#64748b,stroke-width:2px;
    style Platform fill:#0f172a,stroke:#38bdf8,stroke-width:2px;
    style Corp fill:#0f172a,stroke:#a855f7,stroke-width:2px;
    style Dev fill:#0f172a,stroke:#34d399,stroke-width:2px;

SKU restriction policies now lived at MG-Corp and MG-Platform only. MG-Dev had a separate policy set — lighter guardrails, no SKU restrictions, cost caps via DeployIfNotExists budget alerts instead.

Step 4: Drift Detection in CI Pipeline

We built a terraform plan gate into the Policy-as-Code pipeline that runs Resource Graph compliance queries before every apply:

# .github/workflows/policy-deploy.yml
# Validate policy compliance BEFORE apply
- name: Pre-flight Resource Graph Compliance Check
  run: |
    NONCOMPLIANT=$(az graph query -q "
      PolicyResources
      | where type == 'microsoft.policyinsights/policystates'
      | where properties.complianceState == 'NonCompliant'
      | summarize count()
    " --query "data[0].Column1" -o tsv)

    if [ "$NONCOMPLIANT" -gt "10" ]; then
      echo "❌ Pre-flight FAILED: $NONCOMPLIANT non-compliant resources detected."
      echo "Investigate before applying new policy assignments."
      exit 1
    fi
    echo "✅ Pre-flight PASSED: $NONCOMPLIANT non-compliant resources (within threshold)."

Before / After Operational Metrics

Metric	Before	After
Policy Propagation Visibility	Zero (discovered at incident time)	Real-time (Resource Graph + GitHub Actions gate)
BU Onboarding Time	3 weeks manual SOW review	4 hours Terraform module instantiation
Policy Blast Radius (next incident)	Tenant-wide	Scoped to single MG tier

---

Key Takeaway

Azure Policy at the Management Group level is not a UI setting. It is infrastructure. It inherits silently, enforces immediately, and does not warn you about what it breaks before it breaks it.

Treat every policy assignment like a firewall rule change: run a blast radius analysis first, scope the assignment to the narrowest Management Group that satisfies the security requirement, and build a pre-flight compliance query into your CI pipeline before every terraform apply.

If you collapse the CAF Management Group hierarchy to save time during onboarding, you will pay that time back in incident hours. Every time.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin