Terraform builds
the theatre.

Pin your provider versions. An unpinned provider is a time bomb that goes off on a Friday. The azurerm provider is on the 4.x line — note that several argument names changed from 3.x, which is the single most common reason a copied snippet won't apply.

terraform {
  required_version = "~> 1.9"

  required_providers {
    azurerm = { source = "hashicorp/azurerm", version = "~> 4.30" }
    azuread = { source = "hashicorp/azuread", version = "~> 3.0" }
  }

  # remote state: shared, locked, versioned
  backend "azurerm" {
    resource_group_name  = "rg-tfstate"
    storage_account_name = "stcontosotfstate"
    container_name       = "tfstate"
    key                  = "prod/aks.tfstate"
    use_azuread_auth     = true   # no storage keys anywhere
  }
}

provider "azurerm" {
  features {}
  use_oidc = true   # federated auth from CI — no client secret
}

resource "azurerm_virtual_network" "main" {
  name                = "vnet-contoso-prod"
  address_space       = ["10.40.0.0/16"]
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_subnet" "aks" {
  name                 = "snet-aks-nodes"
  resource_group_name  = azurerm_resource_group.main.name
  # ↓ THIS reference is what creates the dependency edge.
  #   No depends_on needed — Terraform reads the graph from your code.
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.40.0.0/20"]
}

resource "azurerm_kubernetes_cluster" "main" {
  name                = "aks-contoso-prod"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "contoso-prod"

  # Standard tier buys a financially-backed control-plane SLA.
  # Free tier gives you an SLO and no money back. Use Standard in prod.
  sku_tier            = "Standard"

  kubernetes_version        = "1.35"   # AKS keeps 3 GA minors in support
  automatic_upgrade_channel = "patch"  # CVEs patch themselves
  node_os_upgrade_channel   = "NodeImage"

  # passwordless everything
  oidc_issuer_enabled       = true
  workload_identity_enabled = true
  local_account_disabled    = true   # no static kubeconfig admin creds

  identity { type = "SystemAssigned" }

  azure_active_directory_role_based_access_control {
    azure_rbac_enabled     = true
    admin_group_object_ids = [var.platform_admins_group_id]
  }

  default_node_pool {
    name                 = "system"
    vm_size              = "Standard_D4as_v5"
    vnet_subnet_id       = azurerm_subnet.aks.id
    auto_scaling_enabled = true     # was enable_auto_scaling in azurerm 3.x
    min_count            = 2
    max_count            = 4
    only_critical_addons_enabled = true  # keep apps off the system pool
    zones                = ["1", "2", "3"]
    upgrade_settings { max_surge = "33%" }
  }

  network_profile {
    # Overlay: pods get IPs from a private space, not your VNet.
    # You stop running out of subnet at 400 pods. Do this.
    network_plugin      = "azure"
    network_plugin_mode = "overlay"
    network_policy      = "cilium"
    network_data_plane  = "cilium"
    pod_cidr            = "192.168.0.0/16"
    load_balancer_sku   = "standard"
  }

  web_app_routing {                     # managed NGINX ingress
    dns_zone_ids = [azurerm_dns_zone.shop.id]
  }

  oms_agent {
    log_analytics_workspace_id      = azurerm_log_analytics_workspace.main.id
    msi_auth_for_monitoring_enabled = true
  }

  lifecycle {
    # the cluster autoscaler owns node_count at runtime — don't fight it
    ignore_changes = [default_node_pool[0].node_count]
  }
}

# app workloads live in their own pool, scaled independently
resource "azurerm_kubernetes_cluster_node_pool" "apps" {
  name                  = "apps"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.main.id
  vm_size               = "Standard_D4as_v5"
  vnet_subnet_id        = azurerm_subnet.aks.id
  auto_scaling_enabled  = true
  min_count             = 3
  max_count             = 30
  zones                 = ["1", "2", "3"]
}

# let the cluster pull images without a single password
resource "azurerm_role_assignment" "acr_pull" {
  scope                = azurerm_container_registry.main.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.main.kubelet_identity[0].object_id
}

Three lines worth arguing about in code review: network_plugin_mode = "overlay" (stops you exhausting VNet IPs), local_account_disabled = true (kills the shared admin kubeconfig that gets pasted into Teams), and ignore_changes on node_count — without it, Terraform and the cluster autoscaler will quietly wrestle each other forever, each one "fixing" the other's work.

The old way: a connection string in a Kubernetes Secret, base64-encoded — which is encoding, not encryption — and which someone will eventually commit. The new way: the pod proves who it is with a short-lived token, and Azure hands it a credential. Nothing to leak, nothing to rotate, nothing to find in a git history two years from now.

# 1. the pod's Azure identity
resource "azurerm_user_assigned_identity" "catalog" {
  name                = "id-catalog-api"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

# 2. the trust: "the catalog-sa account in namespace 'shop' IS this identity"
resource "azurerm_federated_identity_credential" "catalog" {
  name                = "fic-catalog-api"
  resource_group_name = azurerm_resource_group.main.name
  parent_id           = azurerm_user_assigned_identity.catalog.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.main.oidc_issuer_url
  subject             = "system:serviceaccount:shop:catalog-sa"
}

# 3. and what it's allowed to touch — nothing more
resource "azurerm_role_assignment" "catalog_kv" {
  scope                = azurerm_key_vault.main.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.catalog.principal_id
}

On the .NET side that's the whole change — one line, no secret:

builder.Configuration.AddAzureKeyVault(
    new Uri("https://kv-contoso.vault.azure.net/"),
    new DefaultAzureCredential());   // picks up workload identity automatically

One folder per environment, one shared module. Dev and prod differ by a variables file, never by a forked copy of the code — the moment they diverge, "it works in dev" stops meaning anything.

infra/
├── modules/
│   └── aks-platform/        # cluster + network + acr + identities
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
└── envs/
    ├── dev/
    │   ├── main.tf          # module "platform" { source = "../../modules/aks-platform" }
    │   └── dev.tfvars       # node_min=1  node_max=3  sku_tier="Free"
    └── prod/
        ├── main.tf
        └── prod.tfvars      # node_min=3  node_max=30 sku_tier="Standard"

# envs/prod/main.tf
module "platform" {
  source = "../../modules/aks-platform"

  environment   = "prod"
  location      = "westeurope"
  node_min      = 3
  node_max      = 30
  sku_tier      = "Standard"
}

This is the declaration. replicas: 6 is the "what you want." Everything else tells the scheduler how to keep that promise safely.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
spec:
  replicas: 6
  selector:
    matchLabels: { app: catalog-api }
  template:
    metadata:
      labels:
        app: catalog-api
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: catalog-sa      # ← matches the federated credential
      containers:
      - name: api
        image: contosoacr.azurecr.io/catalog-api:1.14.0
        ports: [{ containerPort: 8080 }]
        # requests = what the scheduler reserves. limits = the ceiling.
        resources:
          requests: { cpu: "150m", memory: "192Mi" }
          limits:   { cpu: "800m", memory: "512Mi" }
        # "am I alive?" — fail and I get restarted
        livenessProbe:
          httpGet: { path: /healthz, port: 8080 }
          initialDelaySeconds: 10
        # "am I ready for traffic?" — fail and I'm pulled from the load balancer
        readinessProbe:
          httpGet: { path: /readyz, port: 8080 }
          periodSeconds: 5

The two probes are the most under-used feature in Kubernetes. Liveness answers "should I be killed?" Readiness answers "should I get traffic?" Get readiness wrong and every deploy drops requests, because traffic arrives before your .NET app has finished warming its DI container and building the EF Core model.

In ASP.NET Core the endpoints are three lines:

// Program.cs
builder.Services.AddHealthChecks()
    .AddDbContextCheck<CatalogContext>("db", tags: ["ready"]);

app.MapHealthChecks("/healthz", new() { Predicate = _ => false });      // process is up
app.MapHealthChecks("/readyz",  new() { Predicate = r => r.Tags.Contains("ready") });

Note there is no password anywhere. OIDC federation lets the workflow prove it's this repo, this branch and receive a short-lived Azure token.

name: catalog-api
on:
  push:
    branches: [main]
    paths: ['src/Catalog/**']     # only build what changed

permissions:
  id-token: write            # ← required for OIDC login
  contents: read

env:
  ACR: contosoacr.azurecr.io
  IMAGE: catalog-api

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Test
        run: dotnet test ./src/Catalog --logger trx

      - uses: azure/login@v2
        with:
          client-id:       ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id:       ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: Build & push image
        run: |
          az acr login --name contosoacr
          # tag with the commit SHA — never 'latest' in production
          TAG=${GITHUB_SHA::7}
          docker build -f src/Catalog/Dockerfile \
            -t $ACR/$IMAGE:$TAG .
          docker push $ACR/$IMAGE:$TAG

      - name: Scan image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.ACR }}/${{ env.IMAGE }}:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: '1'          # fail the build, don't just warn

And the Dockerfile — multi-stage, non-root, chiselled runtime. Chiselled images have no shell and no package manager, which removes most of what a scanner would complain about:

FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src
COPY *.sln .
COPY src/Catalog/*.csproj src/Catalog/
RUN dotnet restore src/Catalog          # cached layer: restore before copying code
COPY . .
RUN dotnet publish src/Catalog -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:9.0-noble-chiseled AS final
WORKDIR /app
COPY --from=build /app .
USER $APP_UID                            # non-root by default
ENTRYPOINT ["dotnet", "Contoso.Catalog.dll"]

In GitHub Actions the gate is an environment with protection rules. The job simply names the environment; GitHub does the pausing, the notifying, the audit trail, and the "you can't approve your own deploy" rule. The same idea exists as Approvals and Checks on Azure DevOps environments.

  deploy-canary:
    needs: build
    runs-on: ubuntu-latest
    environment: production-canary      # no reviewers — automatic
    steps:
      - run: az aks get-credentials -g rg-contoso-prod -n aks-contoso-prod
      - run: |
          helm upgrade --install orders-canary ./charts/orders \
            --namespace shop --set image.tag=${GITHUB_SHA::7} \
            --set replicaCount=1 --set canary=true --atomic

      - name: Analyse canary for 10 minutes
        run: ./scripts/canary-analysis.sh --window 10m \
                 --max-error-rate 0.5 --max-p95-regression 20

  deploy-prod:
    needs: deploy-canary
    runs-on: ubuntu-latest
    # ↓ THIS is the human gate. Configured in repo settings:
    #   Required reviewers: @contoso/payments-oncall (2)
    #   Wait timer: 0 · Prevent self-review: on
    #   Deployment branches: main only
    environment:
      name: production
      url: https://shop.contoso.com
    steps:
      - run: |
          helm upgrade --install orders ./charts/orders \
            --namespace shop --set image.tag=${GITHUB_SHA::7} \
            --atomic --timeout 5m
      - run: kubectl rollout status deploy/orders-api -n shop --timeout=180s

Two details that make the gate real rather than decorative. --atomic means a rollout that doesn't go healthy inside the timeout is rolled back automatically — so the approver is authorising an attempt, not a one-way door. And prevent self-review means the person who wrote the change isn't the person who approves it, which is the entire point of having one.

The infrastructure pipeline is deliberately slower

Plan on the pull request so reviewers see the diff. Apply only after a human clicks.

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - run: terraform init
      - run: terraform plan -out=tfplan -lock-timeout=5m
      - run: terraform show -no-color tfplan > plan.txt
      - uses: actions/github-script@v7      # post the plan as a PR comment
        with: { script: '...comment plan.txt on the PR...' }
      - uses: actions/upload-artifact@v4
        with: { name: tfplan, path: tfplan }

  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    environment: prod-infra          # ← required reviewers live here too
    steps:
      - uses: actions/download-artifact@v4
        with: { name: tfplan }
      # apply the SAVED plan, not a fresh one — what was reviewed is what runs
      - run: terraform apply -auto-approve tfplan

Applying the saved plan file matters more than it looks. If you re-plan at apply time, the thing that executes is not the thing anyone reviewed. Drift, a teammate's portal click, or a new provider version can all change it in between — and then your approval was for a document that no longer exists.

spec:
  replicas: 8
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2          # may temporarily run 10
      maxUnavailable: 0    # never dip below 8 ready
  minReadySeconds: 10       # "ready" must hold for 10s before we trust it
  template:
    spec:
      terminationGracePeriodSeconds: 45   # let in-flight requests finish
      topologySpreadConstraints:              # don't put all 8 in one zone
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels: { app: catalog-api }

Scale on the thing that actually predicts load

CPU is a lagging indicator. For orders-api, queue depth is a leading one — KEDA scales on it directly, and can scale to zero when the queue is empty:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: orders-worker }
spec:
  scaleTargetRef: { name: orders-worker }
  minReplicaCount: 1
  maxReplicaCount: 40
  triggers:
  - type: azure-servicebus
    metadata:
      queueName: orders-to-fulfil
      messageCount: "20"       # one pod per 20 queued messages
    authenticationRef: { name: azure-workload-identity }

Protect yourself from your own maintenance

A PodDisruptionBudget stops voluntary disruption — node upgrades, autoscaler consolidation — from taking down more than you can afford. Without it, an AKS node-image upgrade can drain three nodes and briefly leave you with one pod:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: catalog-pdb }
spec:
  minAvailable: "70%"
  selector:
    matchLabels: { app: catalog-api }

And graceful shutdown on the .NET side

Kubernetes sends SIGTERM and waits. If your app exits instantly, you drop whatever was in flight. Generic Host handles this correctly if you let it:

builder.Services.Configure<HostOptions>(o =>
    o.ShutdownTimeout = TimeSpan.FromSeconds(30));   // < terminationGracePeriod

Alert on symptoms your customers feel, not on causes. High CPU is not an incident. A 12% error rate is.

That last one is where the heroes talk to each other: the alert fires because Kubernetes has run out of the room Terraform gave it. The fix is a pull request against max_count, reviewed, planned and applied — not a portal click.