The Pull Request That Blocked $47K/Month in Cloud Spend

“The PR failed in 4 seconds. The problem it caught would have taken 6 months and $280,000 to fully appreciate.”

The Setup

In my current role, I lead the DevSecFinOps practice for enterprise cloud accounts. We run Terraform for Azure Landing Zones — hub-spoke topology, ExpressRoute circuits, Private Endpoints, the full enterprise architecture pattern.

Governance had been aspirational. Policy-as-code was on the roadmap for eighteen months. We had Checkov installed on developer laptops. Nobody ran it consistently.

The cloud spend report was the push. Finance flagged that Azure costs for one account had grown 34% quarter-over-quarter. No new workloads were deployed. The growth was in unused premium resources: idle Public IP Standard SKUs, stranded Premium SSD disks from terminated VMs, and — the expensive one — a zone-redundant Azure Kubernetes Service cluster running at 8% CPU utilisation.

The Mess

The first instinct was to audit the Terraform state manually. That is not an audit — it is an archaeology project.

The AKS cluster had been sized for projected Q3 load. The projection never materialised. The original PR had seven approvals, a thorough code review, and zero comments about cost. Nobody was reviewing infrastructure plans for financial impact. We were reviewing for correctness, not economics.

The team used terraform plan output in PRs. It showed resources being created. It did not show:

• Monthly estimated cost of Standard_D8s_v5 node pools (₹2.2L/month/node × 6 nodes)
• The cost delta between ZoneRedundant and LocallyRedundant storage tiers
• Whether any of the new Public IP Standard SKUs were attached to anything

The pipeline was: write code → peer review → terraform apply → done.

No policy gate. No cost gate. No guardrail between intent and production infrastructure.

---

The Fix: A Three-Layer Pipeline Gate

I built the Checkov + Infracost gate in a single sprint. Three layers, in sequence:

Layer 1: Checkov Security Policy Scan

# .github/workflows/terraform-pr.yml — Security policy gate
- name: Run Checkov IaC Security Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: terraform/
    framework: terraform
    output_format: cli
    soft_fail: false
    skip_check: CKV_AZURE_131 # Internal reasoning documented in SKIP_REASONS.md
    check: >
      CKV_AZURE_4,
      CKV_AZURE_6,
      CKV_AZURE_13,
      CKV_AZURE_16,
      CKV_AZURE_42

The critical checks: public IP exposure (CKV_AZURE_6), TLS version (CKV_AZURE_16), and disk encryption (CKV_AZURE_42). Failures block merge. No exceptions via UI — only documented skips in version control.

Layer 2: Infracost Cost Delta Report

# Cost delta check — must be under $5,000/month increase per PR
- name: Generate Infracost Report
  run: |
    infracost breakdown \
      --path terraform/ \
      --format json \
      --out-file /tmp/infracost.json

    MONTHLY_DELTA=$(cat /tmp/infracost.json | jq '.diffTotalMonthlyCost | tonumber | floor')

    echo "Monthly cost delta: $${MONTHLY_DELTA}"

    if [ "$MONTHLY_DELTA" -gt 5000 ]; then
      echo "❌ COST_GATE_BLOCKED: Delta $${MONTHLY_DELTA}/month exceeds $5,000 threshold"
      echo "Requires FinOps Lead approval before merge."
      exit 1
    fi
    echo "✅ Cost gate passed."

Layer 3: Architecture Compliance (OPA/Conftest)

# policy/azure_aks.rego — Prevents oversized AKS clusters slipping through
package terraform.azure.aks

deny[msg] {
r := input.resource_changes[_]
r.type == "azurerm_kubernetes_cluster_node_pool"

vm_size := r.change.after.vm_size
premium_sizes := {"Standard_D8s_v5", "Standard_D16s_v5", "Standard_E8s_v5"}
vm_size == premium_sizes[_]

node_count := r.change.after.node_count
node_count > 3

msg := sprintf(
"AKS node pool '%s' using premium SKU '%s' with %d nodes requires FinOps approval",
[r.name, vm_size, node_count]
)
}

The Pull Request That Triggered Everything

Three weeks after the gate went live, a PR was raised by a senior engineer to add a new analytics environment. Legitimate, planned work.

The Checkov gate caught a missing disk encryption parameter — fixed in 10 minutes.

Then the Infracost report ran:

Monthly cost delta: $47,239
❌ COST_GATE_BLOCKED: Delta exceeds $5,000 threshold

The PR included a 6-node Standard_D8s_v5 AKS cluster, ZoneRedundant storage for the analytics platform, and a Premium Blob Storage tier for raw data ingestion. All legitimate choices for a production analytics platform. But this was a proof-of-concept.

The engineer didn’t know the PoC sizing would be reviewed differently from a production build. The original ticket didn’t specify.

The PR was revised: LocallyRedundant storage, Standard_D4s_v5 nodes, 3-node pool, Standard_V2 blob tier. Revised monthly cost: $9,200.

$38,000/month saved. Per environment. This team had four non-production environments on the roadmap.

Post-Mortem Finding

The retrospective exposed something more important than the dollar figure: the six months before the gate had twelve Terraform PRs merged. Seven of them had ZoneRedundant resources where LocallyRedundant would have been appropriate. Nobody flagged it because cost was never part of the review template.

Total estimated overspend in that six-month window: $280,000.

The gate cost one sprint to build. The ROI calculation does not require a spreadsheet.

Key Takeaway

A Checkov/Infracost gate without a hard cost threshold is a reporting tool, not a guardrail. The threshold is the policy. Set a number — $2,000, $5,000, whatever your environment warrants — and make the pipeline fail on breach. Require a human FinOps approval to merge. That friction is the point.

The PR review process assumes engineers know the cost implications of their infrastructure choices. Most don’t. The tooling gap is not a knowledge gap — it is a feedback gap. Close the loop at merge time, not at the monthly cloud bill.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin