Project Keystone

Structural — baked into a ConfigMap by the pipeline

The Keystone repo holds plain YAML per service per environment. The deploy pipeline renders it, hashes it, and names the ConfigMap after the hash — so a config change forces a new ConfigMap and therefore a genuine rollout, rather than silently mutating one in place:

# keystone-config/services/pillar/prod.yaml
database:
  commandTimeoutSeconds: 30
  maxPoolSize: 200
endpoints:
  gatehouse: "http://gatehouse.keystone-prod.svc.cluster.local"
telemetry:
  cloudRoleName: "pillar"
  samplingPercentage: 25

# rendered by the pipeline — note the content hash in the name
apiVersion: v1
kind: ConfigMap
metadata:
  name: pillar-config-7f3a91c     # changes when content changes
  namespace: keystone-prod
immutable: true                  # Kubernetes will refuse edits
data:
  appsettings.Production.json: |
    { "Database": { "CommandTimeoutSeconds": 30, ... } }

immutable: true makes Kubernetes reject in-place updates to data, so "somebody edited the ConfigMap at 3am" needs a delete-and-recreate rather than a quiet kubectl edit — visible in an audit log instead of invisible. Not impossible; conspicuous, which is most of the value.

Two things the hash-naming trick needs to actually work. The Deployment's pod spec must be re-rendered to reference the new ConfigMap name — a new ConfigMap nobody references changes nothing, and this is the step teams forget. And because old ConfigMaps accumulate forever, you need a pruning job that skips any still referenced by a retained ReplicaSet, or you'll eventually prune the thing a rollback needed.

Operational — Azure App Configuration with refresh

Read with a sentinel key so the app makes one cheap check rather than watching every setting, and only reloads when the sentinel changes:

// Program.cs — Pillar, Gatehouse and Facade all do this identically
var env = builder.Environment.EnvironmentName;

builder.Configuration.AddAzureAppConfiguration(options =>
{
    options.Connect(new Uri(appConfigEndpoint), new DefaultAzureCredential())
           .Select("Pillar:*", labelFilter: env)
           // Key Vault references resolve with the pod's own identity
           .ConfigureKeyVault(kv => kv.SetCredential(new DefaultAzureCredential()))
           .ConfigureRefresh(refresh =>
           {
               // ← the label MUST match Select's. Register without it and you
               //   watch a null-label sentinel nobody ever bumps: refresh
               //   silently never fires. This is the classic mistake.
               refresh.Register("Pillar:Sentinel", env, refreshAll: true)
                      .SetRefreshInterval(TimeSpan.FromSeconds(30));
           });
},
// ← without optional:true an unreachable store throws at Build()
optional: true);

// the middleware needs this registration or UseAzureAppConfiguration throws
builder.Services.AddAzureAppConfiguration();
var app = builder.Build();
app.UseAzureAppConfiguration();   // drives the refresh check per request

Change ten related values, then bump Pillar:Sentinel once. Every pod picks up all ten together within the refresh interval — no partial application, no deploy.

Secret — Key Vault, referenced

The pod authenticates as itself via workload identity; no secret is ever written to the cluster. In App Configuration the value is a reference, not a literal:

# stored in App Configuration as a Key Vault reference
key   : Pillar:ConnectionStrings:Sql
label : Production
value : {"uri":"https://kv-keystone-prod.vault.azure.net/secrets/pillar-sql-connection"}
type  : application/vnd.microsoft.appconfig.keyvaultref+json

The SDK sees the reference, calls Key Vault with the pod's managed identity, and hands the application a resolved string it never persists. Rotating the secret in Key Vault changes every consumer with no deploy and no config change at all.

Model C still puts a network call on your startup path, so it deserves the same scrutiny we gave model A. Three things make it a materially different risk:

• Snapshots are immutable. A snapshot is a named, frozen subset of the store. It can be created, archived and recovered — but never edited or deleted. Pin a release to a snapshot and you get exactly the reproducibility of baked config, with a managed service doing the serving.
• The provider caches. Configuration is loaded once at startup and then only re-checked on the refresh interval. A blip in the store does not take running pods down; it means they keep serving the values they already have.
• It isn't your uptime. The failure mode is a managed regional service having a bad day, not a two-replica deployment you wrote being stampeded by its own dependents. Those are not the same risk, even when the availability numbers look similar.

The honest caveat: a pod still needs the store on the startup path unless you say otherwise. Layer the providers so structural values come from the baked ConfigMap first and App Configuration improves on them when reachable — and mark it optional, which is the single argument that turns "the store is down so nothing starts" into "the store is down so we run on last-deployed values":

builder.Configuration
    .AddJsonFile("appsettings.json", optional: false)
    .AddJsonFile($"/config/appsettings.{env}.json", optional: true)   // baked ConfigMap
    .AddAzureAppConfiguration(o => { ... }, optional: true);       // wins if reachable

Later providers win, so the cluster can always start. Note that WebApplicationBuilder has already registered appsettings.json and environment variables before you touch it — re-adding them moves them to the end of the chain and changes precedence, which is worth knowing before you copy this into an app that relies on the defaults.

One thing not to claim: an environment variable is not really a break-glass lever here. Changing one means editing the Deployment, which is a rollout — the exact deploy that model B was just criticised for needing.

config/
├── global/                      # affects all three services
│   ├── telemetry.yaml
│   └── resilience.yaml
├── environments/
│   ├── dev.yaml
│   ├── test.yaml
│   └── prod.yaml                # stricter gate on this path
└── services/
    ├── pillar/{dev,test,prod}.yaml
    ├── gatehouse/{dev,test,prod}.yaml
    └── facade/{dev,test,prod}.yaml
schema/
└── config.schema.json           # every file validated against this

A service pipeline is deliberately tiny. It declares what it is and hands everything else to the template — note extends rather than template:

trigger:
  branches: { include: [main] }
  # note the globs: 'docs/*' and '*.md' match one level only
  paths: { exclude: ['docs/**', '**/*.md'] }

resources:
  repositories:
    - repository: templates
      type: git
      name: Keystone/keystone-pipelines
      ref: refs/tags/v3.2.0        # pinned, never a moving branch
  # REDEPLOY (not rebuild) when shared configuration changes. A config-only
  # change must not produce a new image: rebuilding from unchanged source
  # yields a different digest, which is the exact thing Act IV forbids.
  pipelines:
    - pipeline: config
      source: keystone-config-publish
      trigger: { branches: { include: [main] } }

# extends, not template: the service CANNOT add arbitrary steps
extends:
  template: service-pipeline.yml@templates
  parameters:
    serviceName: pillar
    projectPath: src/Pillar.Api/Pillar.Api.csproj
    # selects WHICH pre-configured environment this lands on.
    # The checks on that environment are owned by the platform team.
    riskTier: 1
    runContractTests: true

And the template that defines what a pipeline is allowed to be:

parameters:
  - { name: serviceName,  type: string }
  - { name: projectPath,  type: string }
  - { name: riskTier,     type: number, default: 3 }
  - { name: runContractTests, type: boolean, default: false }

stages:
  # skip the build entirely when the run was started by the config pipeline —
  # we redeploy the digest already in production, with new config beside it
  - stage: build
    condition: ne(variables['Build.Reason'], 'ResourceTrigger')
    jobs:
      - job: build_scan_push
        steps:
          - template: steps/dotnet-build.yml
          - template: steps/unit-test.yml
          - ${{ if eq(parameters.runContractTests, true) }}:
            - template: steps/contract-test.yml
          - template: steps/container-scan.yml   # not optional, by construction
          - template: steps/push-acr.yml

  # one build, three deployments of the SAME digest
  - template: stages/deploy.yml
    parameters: { env: dev,  serviceName: ${{ parameters.serviceName }} }
  - template: stages/deploy.yml
    parameters: { env: test, serviceName: ${{ parameters.serviceName }} }
  - template: stages/deploy.yml
    parameters: { env: prod, serviceName: ${{ parameters.serviceName }} }

Why extends and not a plain template include. An include is a convenience — the calling pipeline can still do whatever else it likes alongside it. An extends template defines the schema of the pipeline: the service repo can only supply the parameters the template accepts. Combine that with a required template check on the production environment and a pipeline that does not extend this template simply cannot deploy to production. The security scan stops being a step people are trusted to include and becomes a property of the system.

The platform repo provisions everything shared. The detail worth copying is the service connection: workload identity federation, so Azure DevOps authenticates to Azure with a short-lived federated token instead of a client secret somebody has to rotate.

# keystone-platform — the pieces the pipelines depend on
resource "azurerm_container_registry" "main" {
  name                = "acrkeystone"
  sku                 = "Premium"
  resource_group_name = azurerm_resource_group.platform.name
  location            = azurerm_resource_group.platform.location
}

# App Configuration data-plane RBAC has NO key or label scoping: Data Reader
# on the store grants read on EVERY key-value in it. Labels are a client-side
# convenience, not a security boundary — so one store PER ENVIRONMENT, or dev
# pods can read production values and production Key Vault reference URIs.
resource "azurerm_app_configuration" "main" {
  for_each            = toset(["dev", "test", "prod"])
  name                = "appcs-keystone-${each.key}"
  sku                 = "standard"   # snapshots + 30-day revision history
  resource_group_name = azurerm_resource_group.platform.name
  location            = azurerm_resource_group.platform.location
}

# the cluster must opt in to workload identity — both are off by default
resource "azurerm_kubernetes_cluster" "main" {
  # ... sku_tier, node pools, network profile ...
  oidc_issuer_enabled       = true
  workload_identity_enabled = true
}

# one identity per (service, environment) — dev and prod must not share one
locals {
  svc_env = {
    for pair in setproduct(["pillar", "gatehouse", "facade"], ["dev", "test", "prod"]) :
    "${pair[0]}-${pair[1]}" => { svc = pair[0], env = pair[1] }
  }
}

resource "azurerm_user_assigned_identity" "service" {
  for_each            = local.svc_env
  name                = "id-${each.key}"
  location            = azurerm_resource_group.platform.location
  resource_group_name = azurerm_resource_group.platform.name
}

# a federated credential carries exactly ONE subject — no wildcards.
# One FIC per namespace, or dev pods fail with AADSTS70021.
resource "azurerm_federated_identity_credential" "service" {
  for_each            = local.svc_env
  name                = "fic-${each.key}"
  resource_group_name = azurerm_resource_group.platform.name
  parent_id           = azurerm_user_assigned_identity.service[each.key].id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.main.oidc_issuer_url
  subject             = "system:serviceaccount:keystone-${each.value.env}:${each.value.svc}"
}

# read its own environment's store — the isolation is the store, not the label
resource "azurerm_role_assignment" "appconfig_read" {
  for_each             = local.svc_env
  scope                = azurerm_app_configuration.main[each.value.env].id
  role_definition_name = "App Configuration Data Reader"
  principal_id         = azurerm_user_assigned_identity.service[each.key].principal_id
}

# without this the Key Vault references resolve to nothing at runtime
resource "azurerm_role_assignment" "kv_read" {
  for_each             = local.svc_env
  scope                = azurerm_key_vault.main[each.value.env].id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.service[each.key].principal_id
}

# and the cluster needs to be able to pull the images it is asked to run
resource "azurerm_role_assignment" "acr_pull" {
  scope                = azurerm_container_registry.main.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.main.kubelet_identity[0].object_id
}

The Kubernetes half is easy to forget and fails silently without it — the ServiceAccount must be annotated with the client id, and the pod labelled to opt in:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pillar
  namespace: keystone-prod
  annotations:
    azure.workload.identity/client-id: "<id-pillar-prod client id>"
---
# and on the pod template
metadata:
  labels:
    azure.workload.identity/use: "true"

Every credential in this system is federated or managed. There is no client secret in a variable group, no connection string in a pipeline, and nothing to rotate at 2am because somebody left. That is not a security nicety — it removes an entire category of incident.

Two things this snippet still leaves open, because they are decisions rather than oversights. Key Vault and App Configuration are shown with default public network access; in a regulated environment you'd put private endpoints in front of both and pay for the DNS complexity. And the platform pipeline itself needs the discipline the Terraform piece argues for: remote state with locking, plan on the pull request, and apply the saved plan rather than a fresh one.

# keystone-pipelines/stages/deploy.yml
parameters:
  - { name: env, type: string }
  - { name: serviceName, type: string }
  # passed in explicitly — variables do NOT flow across stages implicitly.
  # $(imageDigest) set in the build stage renders as empty string here.
  - { name: imageDigest, type: string }

stages:
- stage: deploy_${{ parameters.env }}
  # sequential: two releases never touch the same environment at once
  lockBehavior: sequential
  jobs:
  - deployment: deploy
    # ↓ PER SERVICE, not one shared environment. Environment checks apply to
    #   every pipeline targeting that environment — one shared keystone-prod
    #   would force Facade's daily deploys through Pillar's 2-approver gate.
    #   The ".keystone-${env}" suffix binds the Kubernetes resource.
    environment: keystone-${{ parameters.env }}-${{ parameters.serviceName }}.keystone-${{ parameters.env }}
    strategy:
      runOnce:
        deploy:
          steps:
          # the rendered ConfigMap comes from the CONFIG pipeline, not this one
          - download: config
            artifact: rendered-config

          - task: KubernetesManifest@1
            inputs:
              action: deploy
              namespace: keystone-${{ parameters.env }}
              manifests: |
                $(Pipeline.Workspace)/config/rendered-config/${{ parameters.serviceName }}/*.yaml
                $(Pipeline.Workspace)/manifests/*.yaml
              # digest, never a tag — a tag can be moved, a digest cannot
              containers: acrkeystone.azurecr.io/${{ parameters.serviceName }}@${{ parameters.imageDigest }}

          - task: AzureCLI@2
            displayName: Annotate the release in App Insights
            inputs:
              azureSubscription: sc-keystone-wif   # workload identity federation
              scriptType: bash
              inlineScript: |
                # puts a marker on every latency chart at this exact moment
                az monitor app-insights component update ...

# a SEPARATE stage on the same environment, so its pre-deployment
# Azure Monitor check evaluates AFTER the rollout above has landed.
# This is the only way to get post-deploy verification from a check.
- stage: verify_${{ parameters.env }}
  dependsOn: deploy_${{ parameters.env }}
  jobs:
  - deployment: verify
    environment: keystone-${{ parameters.env }}-${{ parameters.serviceName }}-verify
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo "Alert rules were quiet for the soak window."

Checks configured on the keystone-prod environment, in the UI, by the platform team:

Check	Configuration	Stops
Approval	2 reviewers from Platform; requester may not approve	Shipping unreviewed
Branch control	refs/heads/main only, protection required	Shipping from a feature branch
Business hours	09:00–16:00 Mon–Thu	Friday-evening heroics
Required template	Must extend service-pipeline.yml	Skipping the security scan
Exclusive lock	On; ordering set by lockBehavior in YAML	Two releases colliding
Azure Monitor	No firing alerts on the service's App Insights rules	Deploying onto an already-unhealthy service

Notice that none of these appear in any YAML file a service developer can edit. That separation — pipeline logic in code, deployment permission in the platform — is what makes this governable rather than merely automated. (The one honest exception is the exclusive lock, where the check turns locking on but the runLatest-versus-sequential behaviour is chosen in YAML.)

Risk tiers, and how not to let a service grade its own homework

Because checks bind to environments rather than to pipelines, a single shared keystone-prod would apply Pillar's two-approver gate to Facade's daily deploys. So each service gets its own environment — keystone-prod-facade, keystone-prod-pillar — pre-configured by the platform team with the checks that tier deserves. The riskTier parameter in a service pipeline selects which environment it lands on; it does not define what that environment demands. A team can change their tier and find themselves in a stricter room, but they cannot weaken the room they are standing in.

The break-glass path

A business-hours check that runs 09:00–16:00 is exactly right for planned work and exactly wrong at 02:14, so the design needs a documented way through. Give the on-call rota a dedicated emergency environment with a single-approver check and no business-hours restriction, reachable only from a pipeline run with an explicit reason parameter — and alert on every use. A gate system with no emergency path doesn't get respected; it gets bypassed by someone with permissions nobody remembered they had.

One workspace, four cloud role names, and the trace context flowing across every hop. Distributed tracing is the difference between "the system is slow" and "Gatehouse's call to the payments partner is slow, and here is the exact request":

// identical in all four services; only the role name differs
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddSingleton<ITelemetryInitializer>(
    new CloudRoleNameInitializer(builder.Configuration["Telemetry:CloudRoleName"]));

// the release annotation: makes "which deploy caused this?" a one-glance question
builder.Services.Configure<TelemetryConfiguration>(c =>
    c.TelemetryInitializers.Add(new VersionInitializer(
        Environment.GetEnvironmentVariable("IMAGE_DIGEST"))));

The alert rules that gate the deployment, and that woke somebody at 02:14:

Signal	Threshold	Wired to
Failed request rate	> 2% over 5 min, per role	Pager + the Azure Monitor deployment check
p95 server duration	> 2× the 7-day baseline	Pager
Dependency failures	> 5% to any single partner	Gatehouse team channel
Config refresh failures	Any, sustained 10 min	Platform team — the store is unreachable
Config drift	Store differs from keystone-config main	Platform team — an emergency edit was never back-ported

That last row is the one people forget. If pods silently stop refreshing configuration, nothing breaks immediately — they just quietly drift out of date until the day somebody changes a value and half the fleet doesn't get it.