Clearance Denied: how a request actually gets through Front Door, APIM and the AKS load balancer

At 03:12 UTC a request checks in for a flight it will never remember taking. It is a POST /v1/orders from a mobile app in Pune, roughly 900 bytes of JSON, carrying a bearer token it did not read and a User-Agent string it cannot vouch for. It believes it is going straight to your API. It is not. It is about to walk the full length of an international terminal, and at every desk somebody is going to ask it a question it might not survive.

Most architecture diagrams draw this journey as three friendly boxes and two arrows. That drawing is a lie of omission. Between the client and the pod there are at least five distinct trust boundaries, three different load-balancing algorithms, two TLS terminations, one firewall that runs before routing is even decided, and a Layer 4 appliance that you never explicitly created and probably cannot find in the portal.

So let's walk it properly. Terminal door to aircraft door. One passenger, one bad afternoon.

Here is the shape of the building. Read it once now; the rest of the article is a slow walk through it.

   CLIENT (Pune)
        │  DNS → anycast IP
        ▼
┌────────────────────────────────────────────────────────────────────┐
│  AZURE FRONT DOOR  — global, ~200 PoPs, anycast + split TCP        │
│                                                                    │
│   1. TLS terminated at the nearest PoP                             │
│   2. Host header → which profile is this?                          │
│   3. WAF evaluated  ← happens BEFORE routing and caching           │
│   4. Route match  → 5. Rule set  → 6. Cache lookup                 │
│   7. Origin selection (health probe + latency/priority/weight)     │
└────────────────────────────────────────────────────────────────────┘
        │  Microsoft backbone (not the public internet)
        │  Private Link — the origin has no public IP at all
        ▼
┌────────────────────────────────────────────────────────────────────┐
│  API MANAGEMENT  — regional, the boarding-pass desk                │
│                                                                    │
│   inbound  → check-header X-Azure-FDID                             │
│              validate-jwt                                          │
│              rate-limit-by-key / quota-by-key                      │
│              validate-content (schema)                             │
│   backend  → forward-request, retry, circuit break                 │
│   outbound → strip internal headers, shape the response            │
│   on-error → the only lane that runs when something explodes       │
└────────────────────────────────────────────────────────────────────┘
        │  VNet — private IP, no internet leg
        ▼
┌────────────────────────────────────────────────────────────────────┐
│  AKS — the apron                                                   │
│                                                                    │
│   Azure internal Load Balancer (Standard SKU)                      │
│     frontend IP   10.240.0.25                                      │
│     backend pool  = node NICs / node IPs                           │
│     LB rule       :443 → :30412   +  health probe                  │
│        │                                                           │
│        ▼                                                           │
│   node → kube-proxy (iptables/IPVS/eBPF) → ingress controller      │
│        → Service ClusterIP → POD                                   │
└────────────────────────────────────────────────────────────────────┘

The first thing that happens is not routing. It is geography.

Your custom domain resolves — through a CNAME chain that ends at Front Door — to an anycast address. Anycast means the same IP is announced from every Front Door point of presence on the planet simultaneously, and internet routing hands the client to whichever PoP is topologically nearest. Our Pune request never crosses an ocean to say hello. It shakes hands with a building in India.

That handshake matters more than people expect. TLS negotiation is the expensive part of a short HTTPS request — multiple round trips before a single byte of your JSON moves. Front Door terminates TLS at the PoP, then reuses warm, already-established connections over the Microsoft backbone to your origin. This is "split TCP": the slow, lossy, high-RTT leg is made as short as physically possible, and the long leg runs on a network with no congestion drama.

Only once the connection exists does Front Door look at the Host header to work out whose Front Door profile this request belongs to. This is why an unregistered custom domain fails in a confusing way: the PoP answered the door, but nobody in the building recognises the name on the envelope.

The order of operations, which almost everyone gets wrong

Front Door's processing order is documented and it is not the order most people assume:

The first officer our request meets does not ask for a boarding pass. It does not know what a tenant is. It looks at shape: the URI, the query string, the headers, the body, the source IP's reputation, and the rate at which this client has been arriving.

On Front Door Premium you get the managed rule sets — the Microsoft Default Rule Set (an OWASP Core-Rule-Set derivative) and the Bot Manager rule set — plus custom rules you write yourself. The mental model that keeps people out of trouble:

• Managed rules are the generic threat library. SQL injection, XSS, path traversal, protocol violations, known-bad user agents. You did not write them and you should not try to.
• Custom rules are your business's physics. "Nobody outside these three countries should be hitting /v1/admin." "No client IP gets more than 300 requests per minute to /v1/orders."
• Custom rules are evaluated before managed rules, which is how you carve out an allow-list for the one legacy partner whose payload will otherwise trip rule 942100 forever.

And the rule that saves careers: run in Detection mode first. Every managed rule set will block something legitimate in your estate. You want to discover which one on a Tuesday afternoon in a log query, not at 03:12 UTC in an incident bridge.

resource waf 'Microsoft.Network/FrontDoorWebApplicationFirewallPolicies@2024-02-01' = {
  name: 'wafOrdersApi'
  location: 'Global'
  sku: { name: 'Premium_AzureFrontDoor' }
  properties: {
    policySettings: {
      enabledState: 'Enabled'
      mode: 'Detection'          // (1) flip to 'Prevention' only after a clean week
      requestBodyCheck: 'Enabled'
    }
    managedRules: {
      managedRuleSets: [
        { ruleSetType: 'Microsoft_DefaultRuleSet', ruleSetVersion: '2.1' }
        { ruleSetType: 'Microsoft_BotManagerRuleSet', ruleSetVersion: '1.1' }
      ]
    }
    customRules: {
      rules: [
        {
          name: 'ThrottleOrders'
          priority: 100                // (2) lower number = evaluated first
          ruleType: 'RateLimitRule'
          rateLimitDurationInMinutes: 1
          rateLimitThreshold: 300
          action: 'Block'
          matchConditions: [
            {
              matchVariable: 'RequestUri'
              operator: 'Contains'
              matchValue: [ '/v1/orders' ]
            }
          ]
        }
      ]
    }
  }
}

Two things worth internalising. First, WAF rate limiting is per PoP — it is a blunt instrument for stopping floods, not a precise instrument for enforcing a customer's contractual quota. That job belongs one checkpoint later, in APIM, where identity actually exists. Second, if you are behind Front Door, the client IP the WAF sees is the real one; the IP your pod sees will not be, and we will come back to that in section 07.

The WAF asked what does this look like. APIM asks who are you, what are you entitled to, and have you had too much already today. It is the first component in the chain with a concept of identity, product, subscription and quota.

Everything APIM does to a request happens in a four-stage pipeline, and understanding the stages is most of understanding the product:

  request ──▶ inbound ──▶ backend ──▶ [ your API in AKS ]
                 │           │                    │
                 │           └──── response ◀──────┘
                 │                 │
                 │                 ▼
                 │            outbound ──▶ client
                 │
                 └── throw ──▶ on-error ──▶ client
                     (remaining steps in the current
                      section are skipped entirely)

Policies are inherited down five scopes — global → workspace → product → API → operation — and the <base /> element is where the parent's policies get spliced in. Put <base /> first in each section unless you have a specific reason not to; the day you forget, an API-scope policy silently deletes your global security posture for that one API and nothing turns red.

The inbound section, written like you mean it

<policies>
  <inbound>
    <base />

    <!-- (1) Did you come through the front door, or over the fence? -->
    <check-header name="X-Azure-FDID"
                  failed-check-httpcode="403"
                  failed-check-error-message="Invalid request."
                  ignore-case="false">
      <value>{{FrontDoorId}}</value>
    </check-header>

    <!-- (2) Identity, verified against the issuer's own keys -->
    <validate-jwt header-name="Authorization"
                  failed-validation-httpcode="401"
                  failed-validation-error-message="Unauthorized">
      <openid-config url="https://login.microsoftonline.com/{{tenantId}}/v2.0/.well-known/openid-configuration" />
      <audiences><audience>api://orders</audience></audiences>
      <required-claims>
        <claim name="roles" match="any"><value>Orders.Write</value></claim>
      </required-claims>
    </validate-jwt>

    <!-- (3) Quota per *caller*, not per PoP. This is the real contract. -->
    <rate-limit-by-key calls="120" renewal-period="60"
        counter-key="@(context.Subscription?.Id ?? context.Request.IpAddress)" />
    <quota-by-key calls="100000" renewal-period="86400"
        counter-key="@(context.Subscription.Id)" />

    <!-- (4) Reject malformed bodies here, not in your domain layer -->
    <validate-content unspecified-content-type-action="prevent"
                      max-size="102400" size-exceeded-action="prevent"
                      errors-variable-name="bodyErrors">
      <content type="application/json" validate-as="json" action="prevent" />
    </validate-content>

    <!-- (5) Give the pod a thread to pull on -->
    <set-header name="X-Correlation-Id" exists-action="skip">
      <value>@(context.RequestId.ToString())</value>
    </set-header>
  </inbound>

  <backend><base /></backend>

  <outbound>
    <base />
    <!-- (6) The pod's opinions are not the client's business -->
    <set-header name="X-Powered-By" exists-action="delete" />
    <set-header name="Server" exists-action="delete" />
  </outbound>

  <on-error>
    <base />
    <set-body>@{ return new JObject(
        new JProperty("error","upstream_failure"),
        new JProperty("correlationId", context.RequestId)).ToString(); }</set-body>
  </on-error>
</policies>

Line (1) is the one people skip, and it is the one that matters most. X-Azure-FDID is a GUID unique to your Front Door profile, injected by Front Door itself. Without that check, anyone who discovers your APIM hostname can bypass the WAF entirely by talking to APIM directly — the entire edge tier becomes decorative. Microsoft's own guidance is to do both this and an IP restriction using the AzureFrontDoor.Backend service tag.

Here is the failure mode that survives most security reviews: the chain is perfect, the WAF is in Prevention, the JWT validation is airtight — and the AKS ingress still has a public IP with a permissive NSG, because that is how it was during the spike three sprints ago.

The fix is to make the origin unreachable from the internet by construction rather than by policy. Front Door Premium supports Private Link origins, and the supported origin list includes internal load balancers — explicitly naming AKS — as well as App Service, Blob Storage, Application Gateway, Container Apps and API Management.

The flow is deliberately two-sided, and that is the point:

1. You enable Private Link on the origin in your Front Door profile.
2. Front Door creates a private endpoint on your behalf, from its own managed regional network.
3. A pending connection request appears on your resource. Nothing flows until you approve it — portal, CLI or PowerShell.
4. A few minutes later the tunnel is live. Traffic now goes client → global PoP → Microsoft backbone → Front Door's regional cluster → private endpoint → your origin.

Two constraints to plan around. Front Door Private Link is only offered from regions with Availability Zone support; if your origin sits elsewhere, traffic hops in from the nearest supported region and you pay for that in latency. And approval is a human-in-the-loop step, which means it belongs in your runbook, not in your Bicep deployment's happy path.

When Private Link is not on the table, the belt-and-braces version is a layered lockdown:

This is the part of the chain that surprises people, because it is the part where a Kubernetes object silently becomes an Azure object.

You write this:

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "apps-subnet"
    service.beta.kubernetes.io/azure-load-balancer-ipv4: 10.240.0.25
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - port: 443
      targetPort: 8443
  selector:
    app: ingress-nginx

And this happens, without you asking:

 kubectl apply
      │
      ▼
 kube-apiserver  ──▶ Service object created, status.loadBalancer = {} (pending)
      │
      ▼
 cloud-controller-manager  (the Azure cloud provider, running in the cluster)
      │
      ├─▶ finds or creates the Standard SKU Azure Load Balancer
      │      kubernetes-internal   (in the MC_* node resource group)
      │
      ├─▶ frontend IP configuration
      │      private IP 10.240.0.25 from apps-subnet
      │      (or, for a public Service, a brand-new Public IP —
      │       each Service gets its own dedicated frontend IP)
      │
      ├─▶ backend pool
      │      every eligible node NIC / node IP in the cluster
      │      updated automatically as node pools scale
      │
      ├─▶ load balancing rule
      │      frontend 10.240.0.25:443  ─▶  backend :nodePort (30000-32767)
      │
      ├─▶ health probe
      │      externalTrafficPolicy: Local   → probes spec.healthCheckNodePort
      │      externalTrafficPolicy: Cluster → probes the service port (TCP by default)
      │
      └─▶ NSG rules on the node subnet, opened to match the LB rule
      │
      ▼
 Service.status.loadBalancer.ingress = 10.240.0.25   ← now EXTERNAL-IP populates

A handful of details worth carrying around in your head:

• The load balancer lives in the node resource group (MC_<rg>_<cluster>_<region>), which is why people cannot find it. Do not hand-edit anything in there; the controller will reconcile your change away, usually at the least convenient moment.
• The SKU is immutable after cluster creation, and one cluster gets one SKU. Standard is the default and the only sane choice — it is what gives you availability zones, larger backend pools and secure-by-default behaviour.
• Every type: LoadBalancer Service gets its own frontend IP on the same LB. Twenty Services means twenty frontend IPs and twenty rules, which is precisely why you want one Service of type LoadBalancer fronting an ingress controller, and everything else behind it as ClusterIP.
• spec.loadBalancerIP is deprecated. Use the azure-load-balancer-ipv4 / -ipv6 annotations, and make sure the address is in the cluster's VNet and unclaimed.

The annotations that are actually worth knowing

The Azure load balancer is Layer 4. It does not know what a pod is. It picks a node and hands the connection over, and from that moment the request is Kubernetes' problem.

What happens next depends entirely on one field you probably left at its default.

 externalTrafficPolicy: Cluster   (the default)
 ─────────────────────────────────────────────────────────────
   LB ──▶ node-3 ──(kube-proxy SNATs, then forwards)──▶ node-7 ──▶ pod
                        │
                        └── source IP rewritten to node-3
                            extra network hop
                            even spread across all nodes
                            no imbalance if pods are uneven

 externalTrafficPolicy: Local
 ─────────────────────────────────────────────────────────────
   LB ──▶ node-3 ──▶ pod on node-3 (only)
                        │
                        └── client source IP preserved
                            one hop, lower latency
                            nodes with no pod fail the probe and get
                            excluded — traffic skews to pod-dense nodes

The mechanism is worth spelling out, because it explains a class of bug that looks like magic. With Local, Kubernetes allocates a spec.healthCheckNodePort, and the Azure health probe targets that port rather than your application. The node answers healthy only if it is hosting at least one ready endpoint for the Service. Nodes without a pod fail the probe and drop out of rotation — which is exactly how the "only send traffic where the pod actually is" guarantee is implemented. With Cluster and shared health probe mode enabled, the probe instead targets kube-proxy on the healthCheckNodePort for all cluster-policy Services at once, which is how large clusters avoid drowning in per-Service probes.

And the client IP question. By the time our Pune request reaches the pod, it has been through two proxies and possibly one SNAT. The real client address is not in the TCP source; it is in headers:

In ASP.NET Core this means ForwardedHeadersMiddleware is not optional — and it must be configured with known proxies or networks, or you have built an IP-spoofing endpoint and called it observability.

builder.Services.Configure<ForwardedHeadersOptions>(o =>
{
    o.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
    o.ForwardLimit     = 2;                          // APIM + ingress
    o.KnownNetworks.Clear();                        // (1) don't trust the defaults
    o.KnownProxies.Clear();
    o.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("10.240.0.0"), 16));
});

var app = builder.Build();
app.UseForwardedHeaders();                          // (2) FIRST. Before everything.

One more airside option

An Azure Load Balancer plus an in-cluster ingress controller is the workhorse pattern, but it is not the only one. Application Gateway for Containers moves L7 routing, TLS and WAF out of the cluster and into a managed Azure resource driven by Gateway API resources — which is worth considering when you want WAF enforcement at the regional tier as well as the edge, or when you would rather not run and patch an ingress controller. The trade is the usual one: less to operate, less to tune, and a dependency on Azure's release cadence rather than your own Helm chart.

Everything so far has been about traffic coming in. The same load balancer is doing a second job on the way out, and this is where clusters die quietly.

By default AKS creates one managed outbound public IP, and sets allocatedOutboundPorts to 0, which means "allocate automatically based on backend pool size":

Read that table again with an autoscaler in mind. Scaling from 50 nodes to 51 halves the outbound port budget on every node in the cluster, simultaneously. Your pods start failing to reach the database, the token endpoint, the payment gateway — and the trigger was a scale-out, which is the exact opposite of what everyone in the incident channel is looking for.

The arithmetic is fixed and simple: each outbound public IP provides 64,000 ephemeral ports. So

   max nodes = ( 64,000 / ports-per-node ) × number-of-outbound-IPs

   example:   4,000 ports per node,  7 outbound IPs
              ( 64,000 / 4,000 ) × 7  =  112 nodes

az aks update \
  --resource-group $RG --name $CLUSTER \
  --load-balancer-managed-outbound-ip-count 7 \
  --load-balancer-outbound-ports 4000 \
  --load-balancer-idle-timeout 4          # default is 30 minutes

Three habits that keep this boring:

• Pin the port allocation explicitly and size it for your maximum node count, including maxSurge headroom during upgrades. Automatic allocation is a footgun with a scale trigger.
• Drop the idle timeout from the 30-minute default to something like 4 minutes if your workload opens many short-lived outbound connections. Ports are reclaimed only when the flow goes idle for that long.
• Or sidestep SNAT entirely — use outboundType: userDefinedRouting and route egress through Azure Firewall or a NAT Gateway. NAT Gateway in particular has a far more forgiving port model and is the right answer for connection-heavy workloads.

Symptoms to recognise: intermittent outbound connection timeouts, failures that correlate with node count rather than with traffic, and a SnatConnectionCount/AllocatedSnatPorts metric that looks fine on average and terrible at p99.

A five-checkpoint architecture is only debuggable if every checkpoint writes down what it did and every log shares a key. Here is the minimum viable paper trail.

The single highest-leverage move: propagate one identifier from the edge all the way to the pod, and make sure it appears in every one of those six tables. Front Door's X-Azure-Ref is a reasonable root because it is generated before anything else in your system exists, and it is also the value Microsoft support will ask for. Stamp it into an APIM header, log it in the ingress controller's format string, attach it as a baggage item in OpenTelemetry, and the question "where did this request die?" becomes one query instead of six.

This chain is not free. Front Door Premium plus APIM Premium plus a dedicated ingress tier is a real monthly number, and every hop adds latency, configuration surface and a new place to be wrong at 3am. Some honest guidance:

And three anti-patterns that show up in almost every review:

• The decorative edge. Front Door in front, and an origin that still answers the public internet. If X-Azure-FDID isn't checked and the NSG isn't locked to AzureFrontDoor.Backend, the WAF is a suggestion.
• Twenty LoadBalancer Services. Twenty frontend IPs, twenty rules, twenty probes, and an NSG nobody can read. One Service of type LoadBalancer, one ingress controller, everything else ClusterIP.
• Rate limiting in exactly one place. The WAF limits per PoP and knows nothing about tenants; APIM limits per subscription and knows nothing about volumetric floods. They are different tools solving different problems, and you need both.

Our request from Pune made it. One hundred and eighty-four milliseconds, terminal door to pod, and in that time it was fingerprinted by a firewall in Chennai, had its token verified against Entra ID's published keys, had its quota decremented against a subscription it does not know it belongs to, crossed a private endpoint that has no route to the public internet, was handed to a Layer 4 load balancer nobody in your organisation deliberately created, and was steered to a node that had proved five seconds earlier that it was still holding a ready pod.

It will never know any of that happened. That is the whole point. The measure of a good security chain is not how visible it is — it is how completely invisible it is to everything that has legitimate business getting through, and how absolutely final it is for everything that does not.

The passenger boards. The gate closes. Somewhere in a log table, six rows share a tracking reference.

Further reading

• Front Door routing architecture — the definitive order of operations.
• Front Door in front of API Management — including the check-header policy.
• Secure your origin with Private Link — supported origin types and the approval flow.
• AKS Standard load balancer and internal load balancer.
• cloud-provider-azure LoadBalancer annotations — the full annotation reference.
• Configuring outbound ports and SNAT.
• APIM networking options by tier.