Pod Topology Spread Constraints Deep Dive for the CNCF Exams (CKA & CKAD): maxSkew Math, minDomains, matchLabelKeys & Scheduler Defaults

Most candidates meet topology spread constraints as a footnote at the end of the affinity chapter: “if you want replicas balanced across zones, use maxSkew: 1 instead of anti-affinity.” That is true, and our taints, tolerations and affinity guide for the CNCF exams covers the basic shape of the field. But the exams, and real clusters, go further than the one-liner. Why does a six-replica Deployment with maxSkew: 1 across three zones end up 3/2/1 after a rolling update? Why does a Pod sit Pending when there is obviously free capacity? Why does scaling down leave everything in one zone? Why does a brand-new cluster already spread Pods a little, even though you never wrote a constraint?

Every one of those answers lives in the parts of the API the footnote skips: the actual skew calculation, minDomains, matchLabelKeys, the two node-inclusion policies, and the cluster-level default constraints baked into kube-scheduler. This guide teaches those mechanics at the level the CKA (Workloads & Scheduling, 15%) and CKAD (Application Deployment and Pod design tasks) expect, with the kubectl workflow you need to verify and debug them under time pressure.

Topology spread is a KubeAstronaut-wide topic: CKA tests you on placing and troubleshooting it, CKAD on writing it into Deployments, and KCNA conceptually asks why you would spread at all. We treat it once, properly, here.

What a Topology Spread Constraint Actually Says

A constraint lives in pod.spec.topologySpreadConstraints and is evaluated by the PodTopologySpread scheduler plugin at scheduling time. The full field set:

spec:
  topologySpreadConstraints:
  - maxSkew: <integer>                 # required, > 0
    topologyKey: <string>              # required, a node label key
    whenUnsatisfiable: <string>        # DoNotSchedule (default) | ScheduleAnyway
    labelSelector: <object>            # which Pods count toward the spread
    minDomains: <integer>              # optional; GA since v1.30
    matchLabelKeys: <list>             # optional; beta, on by default since v1.27
    nodeAffinityPolicy: [Honor|Ignore] # optional; GA since v1.33
    nodeTaintsPolicy: [Honor|Ignore]   # optional; GA since v1.33

Three vocabulary terms make the rest of the guide readable:

A topology domain is the set of nodes sharing the same value of topologyKey. With topology.kubernetes.io/zone, each zone is a domain. With kubernetes.io/hostname, each node is its own domain.
Matching Pods are the Pods in the same namespace as the incoming Pod that match labelSelector (plus, with matchLabelKeys, the extra keys described below).
Skew for a domain is matching Pods in that domain − global minimum, where the global minimum is the smallest number of matching Pods in any eligible domain, or 0 if there are fewer eligible domains than minDomains.

The scheduler only lets a Pod land on a node if, after placement, the skew of that node’s domain stays ≤ maxSkew (for DoNotSchedule), or it scores nodes to minimise skew (for ScheduleAnyway). Everything else in this article is a consequence of that one rule.

The Skew Calculation, Worked by Hand

Exam questions rarely ask you to recite the definition. They give you a cluster state and ask where the next Pod can go, or why it cannot go anywhere. You need to be able to do the arithmetic in your head.

Example 1: the happy path

Three zones (a, b, c), a Deployment with maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule, and currently 2 matching Pods in a, 2 in b, 1 in c.

Zone	Matching Pods	Skew if next Pod lands here
a	2	3 − 1 = 2 (violates `maxSkew: 1`)
b	2	3 − 1 = 2 (violates)
c	1	2 − 1 = 1 (allowed)

Global minimum is 1 (zone c). The only zone where the resulting skew stays within 1 is c. The scheduler will place the Pod in zone c even if a and b have far more free CPU. Note what this does not say: it never says “put one Pod in every zone”. It says “the gap between the fullest and emptiest domain may not exceed 1”.

Example 2: why a Pod goes Pending

Same constraint, but zone c has no node with free resources (or every node in c is cordoned). Matching Pods: a = 2, b = 2, c = 1. Zone c still counts in the skew calculation because it still has nodes carrying the zone label, so the global minimum stays at 1. Placing in a or b would produce a skew of 2. Result: the Pod stays Pending with an event like:

0/6 nodes are available: 2 node(s) didn't match pod topology spread constraints,
2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: },
2 Insufficient cpu. preemption: 0/6 nodes are available: ...

This is the classic trap: DoNotSchedule is a hard constraint, so it prefers an unschedulable Pod over an unbalanced cluster. If the business requirement is “balance when possible, but always run”, the right answer is whenUnsatisfiable: ScheduleAnyway, not a bigger maxSkew.

Example 3: multiple constraints are ANDed

You can list several constraints. All of them must be satisfiable for a node to be feasible:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels: { app: api }
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels: { app: api }

Read this as: “never let one zone get more than one Pod ahead of another (hard), and prefer spreading across nodes within a zone (soft)”. Combining two hard constraints can create a state where no node satisfies both, which is why the per-node soft constraint is the common pattern. The docs call this out explicitly: conflicting hard constraints leave Pods Pending.

Node Labels: the Part Candidates Forget to Check

A constraint on topology.kubernetes.io/zone is meaningless on nodes that do not carry that label, and the scheduler silently skips nodes that lack the topology key when computing domains. On a cloud-managed cluster the cloud provider sets these well-known labels:

Label	Set by	Typical value
`kubernetes.io/hostname`	kubelet	node name
`topology.kubernetes.io/zone`	cloud provider	`eu-west-1a`
`topology.kubernetes.io/region`	cloud provider	`eu-west-1`

In an exam environment built with kubeadm or kind there is no zone label. If a task says “spread across zones”, your first command is:

kubectl get nodes -L topology.kubernetes.io/zone,kubernetes.io/hostname

If the ZONE column is empty, either the task expects you to add labels (kubectl label node node01 topology.kubernetes.io/zone=zone-a) or it expects kubernetes.io/hostname as the key. A constraint against a label no node has is not an error; it simply never restricts anything, which is a quiet way to fail a task that is later checked by a grader counting Pods per zone.

minDomains: Spreading Before the Domains Exist

The global minimum is “the smallest count of matching Pods in any eligible domain”. That definition has a hole: when the cluster currently has only two zones but you know a third will be added by the autoscaler, the scheduler cannot “reserve” capacity for the missing zone, because it only learns about domains from existing nodes.

minDomains closes the hole. Rules from the API:

Must be greater than 0 when set.
Only valid with whenUnsatisfiable: DoNotSchedule.
If the number of eligible domains with the topology key is less than minDomains, the global minimum is treated as 0.
If the number of eligible domains is ≥ minDomains, the field has no effect.
Unset behaves like minDomains: 1.

Worked example: maxSkew: 1, minDomains: 3, but only zones a and b exist, with 1 Pod each. Normally the global minimum would be 1 and a third Pod could land in either zone (skew 2 − 1 = 1). With minDomains: 3, the global minimum is forced to 0, so placing a second Pod anywhere would make skew 2 − 0 = 2 > 1, and the Pod goes Pending until a third zone appears. That is exactly what you want when a cluster autoscaler reacts to Pending Pods by provisioning nodes in the missing zone, and exactly what you do not want on a fixed-size exam cluster. If a question describes “Pods remain Pending although two zones have capacity”, minDomains is a strong candidate explanation.

matchLabelKeys: Fixing the Rolling-Update Skew

This is the subtlety most tutorials miss and the one most likely to surprise you in production. A Deployment’s labelSelector typically matches app: web. During a rolling update, both the old ReplicaSet’s Pods and the new ReplicaSet’s Pods carry app: web, so they are counted together. The scheduler happily balances the total, which means the new revision can end up heavily skewed while the old revision drains, and the final state after the old Pods disappear is unbalanced.

matchLabelKeys lets you say “count only Pods that share these label values with the incoming Pod”. The canonical use is pod-template-hash, which the Deployment controller stamps on every Pod of a given ReplicaSet:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: web
  matchLabelKeys:
  - pod-template-hash

Now the constraint effectively becomes “spread the Pods of this revision”, and each rollout balances independently. Rules worth memorising:

A key cannot appear in both matchLabelKeys and labelSelector.
matchLabelKeys requires labelSelector to be set.
Keys are read from the incoming Pod’s labels at scheduling time; editing a running Pod’s labels later does not re-evaluate anything.
Since v1.34 the matching key/value pairs are explicitly merged into labelSelector (previously handled implicitly). The behaviour you observe is the same; the API object you read back is more explicit.

The alternative without matchLabelKeys is to change the Pod template’s labels on each release (for example a version label) and reference that in the selector. That works but means editing the spec every rollout. On an exam, if a task says “ensure each rollout is balanced independently”, matchLabelKeys: [pod-template-hash] is the one-line answer.

nodeAffinityPolicy and nodeTaintsPolicy: Which Nodes Count?

Both fields answer the same question: when calculating skew, which nodes belong to the set of eligible domains?

Field	Default	`Honor`	`Ignore`
`nodeAffinityPolicy`	`Honor`	Only nodes matching the Pod’s `nodeSelector` / `nodeAffinity` are included	All nodes included regardless of the Pod’s affinity
`nodeTaintsPolicy`	`Ignore`	Tainted nodes are included only if the Pod tolerates the taint	Tainted nodes are included unconditionally

Why this matters: suppose your Pod has nodeSelector: {disktype: ssd} and only zones a and b have SSD nodes. With the default nodeAffinityPolicy: Honor, zone c is not an eligible domain, so the global minimum is computed over a and b only and the Pod is not held hostage by an empty zone it could never use. If you set Ignore, zone c counts, its 0 matching Pods become the global minimum, and your constraint may be impossible to satisfy.

The taints policy defaults the other way. Tainted nodes (say, a gpu=true:NoSchedule pool) are included in the domain set by default even if the Pod does not tolerate them. If that produces phantom domains that drag the global minimum down, set nodeTaintsPolicy: Honor. Both fields graduated to GA in v1.33, so you can rely on them in any current exam environment.

The Defaults You Never Wrote: Cluster-Level Constraints

A frequent “why does my cluster already spread Pods?” question. If kube-scheduler is not given explicit defaultConstraints, it behaves as though every Pod had these (stable since v1.24):

defaultConstraints:
- maxSkew: 3
  topologyKey: "kubernetes.io/hostname"
  whenUnsatisfiable: ScheduleAnyway
- maxSkew: 5
  topologyKey: "topology.kubernetes.io/zone"
  whenUnsatisfiable: ScheduleAnyway

Key behaviours:

They are soft (ScheduleAnyway), so they only influence scoring; they never leave a Pod Pending.
They apply only to Pods that define no topologySpreadConstraints of their own and that belong to a Service, ReplicaSet, StatefulSet or ReplicationController, because the selector is inferred from those owners.
The plugin does not score nodes that lack the topology key, so on a cluster without zone labels only the hostname rule has any effect.
The legacy SelectorSpread plugin, which did similar work, is disabled by default.

A cluster administrator can replace them in a KubeSchedulerConfiguration:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  pluginConfig:
  - name: PodTopologySpread
    args:
      defaultConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      defaultingType: List

defaultingType: List means “use exactly my list”; System means “use the built-in defaults”. To disable defaults entirely, set defaultingType: List with an empty defaultConstraints: []. This is CKA territory: the scheduler config file is referenced from the static Pod manifest in /etc/kubernetes/manifests/kube-scheduler.yaml via --config, and editing it restarts the scheduler, which our static Pods guide covers.

Topology Spread vs Pod Anti-Affinity: Choosing on the Exam

The affinity guide already gives the short answer (spread is cheaper and expresses balance, anti-affinity expresses exclusion). The comparison below is the exam-decision version:

Requirement in the task	Use	Why
”At most one replica per node”	`requiredDuringScheduling` pod anti-affinity on `kubernetes.io/hostname`	Anti-affinity is a strict per-domain cap of 1
”Replicas evenly across zones, never more than 1 apart”	`topologySpreadConstraints`, `maxSkew: 1`, `DoNotSchedule`	Expresses skew, allows multiple per domain
”Prefer spreading but always schedule”	`ScheduleAnyway`	Soft scoring, no `Pending` risk
”Balance each rollout independently”	add `matchLabelKeys: [pod-template-hash]`	Anti-affinity has no equivalent
”Keep Pods away from a specific other app”	pod anti-affinity with that app’s selector	Spread only counts Pods matching the same selector
”Spread, but only among SSD nodes”	spread + `nodeSelector` (default `nodeAffinityPolicy: Honor`)	Domain set respects affinity

A pattern that works in practice: anti-affinity (soft) for “don’t co-locate with my own replicas on one node” plus a hard zone spread. It costs little and covers both node loss and zone loss.

Known Limitations and the Scale-Down Trap

The documentation lists limitations that turn into exam distractors:

No rebalancing after removal. Constraints are checked only at scheduling time. Scale a Deployment from 6 to 3 and the ReplicaSet controller picks victims by its own rules, which can leave all 3 survivors in one zone. The fix is an external tool such as the Descheduler, which evicts Pods that violate RemovePodsViolatingTopologySpreadConstraint so they get rescheduled. Nothing in core Kubernetes does this for you.
Domains are discovered from existing nodes. A node pool scaled to zero is invisible, which is the minDomains use case above.
Tainted-node matching Pods are respected. Pods already running on tainted nodes count toward skew even if the incoming Pod cannot land there.
Ghost Pods. If a Pod’s own labels do not match its constraint’s labelSelector, it does not count itself. Several such Pods can pile into one domain because none of them changes the calculation. Always make the Pod match its own selector; an exam task that gives you app: web in the template and asks you to write a selector for app: frontend is testing exactly this.

Debugging a Spread Constraint in 60 Seconds

When a Pod is Pending and you suspect spread:

# 1. Confirm the reason
kubectl describe pod web-7d9c4-x2k1q | grep -A3 Events

# 2. See the actual distribution of matching Pods by domain
kubectl get pods -l app=web -o wide --sort-by=.spec.nodeName
kubectl get pods -l app=web -o custom-columns=\
'NAME:.metadata.name,NODE:.spec.nodeName' --no-headers | \
while read p n; do kubectl get node "$n" -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}'; done | sort | uniq -c

# 3. Check the labels the constraint depends on
kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pod web-7d9c4-x2k1q -o jsonpath='{.spec.topologySpreadConstraints}' | jq

# 4. Check whether a cordon or taint shrank the usable domain
kubectl get nodes -o custom-columns='NAME:.metadata.name,SCHED:.spec.unschedulable,TAINTS:.spec.taints[*].key'

Step 2 is the one most candidates skip, and the one that reveals a 3/3/0 distribution that makes the Pending state obvious. For the jsonpath and custom-columns syntax used here, see our kubectl output formatting guide.

Common exam fixes, in order of how often they are the intended answer:

Change whenUnsatisfiable to ScheduleAnyway.
Fix the labelSelector so it matches the Pod’s own labels.
Add the missing node label the task assumed exists.
Uncordon or untaint the node that shrank the domain (see the drain, cordon and PDB guide).
Raise maxSkew only if the task explicitly allows a looser balance.

A Complete Production-Grade Manifest

Everything in one Deployment, annotated:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: shop
spec:
  replicas: 9
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      topologySpreadConstraints:
      # Hard: zones never more than 1 apart, per rollout revision
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        minDomains: 3
        labelSelector:
          matchLabels:
            app: api
        matchLabelKeys:
        - pod-template-hash
        nodeTaintsPolicy: Honor
      # Soft: spread across nodes inside each zone
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: api
      containers:
      - name: api
        image: registry.example.com/shop/api:2.4.1
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"

Nine replicas across three zones gives 3/3/3; a rollout keeps each revision balanced; node loss inside a zone is softened by the hostname rule; and minDomains: 3 makes the scheduler wait for a third zone rather than pile into two if the cluster is still scaling up. Pair this with a PodDisruptionBudget so that voluntary disruptions respect the same availability goal.

How the Exams Test This

CKA tasks phrase it operationally: “Deployment web in namespace prod must have its Pods spread evenly across nodes; some Pods are Pending, fix the configuration without changing replica count.” The grader counts Pods per node afterwards. Expect to inspect node labels, read the constraint, and often switch DoNotSchedule to ScheduleAnyway or correct a selector. Occasionally a task is about the scheduler config itself.

CKAD tasks phrase it declaratively: “Create a Deployment with 4 replicas spread across zones with a maximum skew of 1 using the topology.kubernetes.io/zone key.” You are graded on the YAML. The time-saver is generating the Deployment with kubectl create deployment --dry-run=client -o yaml and inserting the topologySpreadConstraints block under spec.template.spec, which is the exact indentation level candidates get wrong most often (it is a sibling of containers, not a child of it).

KCNA asks conceptually: which scheduling mechanism balances replicas across failure domains? Answer: topology spread constraints, with maxSkew describing allowed imbalance.

Because the scenarios are hands-on and timed, the fastest preparation is repeated practice in a cluster where you can break and fix spread constraints yourself. The KubeAstronaut mock exam bundle covers scheduling across the CKA, CKAD and KCNA question styles in one path, so you drill the same mechanic from the three angles the exams take. If you are preparing for a single exam, the CKA bundle and CKAD bundle focus on their own task formats. For a free local lab to try everything in this guide, see how to set up a Kubernetes practice lab for the CNCF exams.

Frequently Asked Questions

What is maxSkew in Kubernetes topology spread constraints?

maxSkew is the maximum allowed difference between the number of matching Pods in any topology domain and the global minimum across eligible domains. With DoNotSchedule the scheduler refuses placements that would exceed it; with ScheduleAnyway it uses skew to score nodes and prefers the lowest.

Why are my Pods Pending even though nodes have free resources?

With whenUnsatisfiable: DoNotSchedule, the only feasible nodes are those in domains whose resulting skew stays within maxSkew. If those domains are full, cordoned, or tainted, the Pod waits. Check the distribution per domain, the node labels, and whether minDomains is forcing the global minimum to zero.

What is the difference between DoNotSchedule and ScheduleAnyway?

DoNotSchedule is a hard filter: violating nodes are removed from consideration and the Pod may stay Pending. ScheduleAnyway is a soft preference: the scheduler still places the Pod but scores nodes to minimise skew.

When should I use matchLabelKeys?

Add matchLabelKeys: [pod-template-hash] on Deployments so each rolling-update revision is balanced independently instead of being counted together with the Pods of the previous revision.

Do topology spread constraints rebalance Pods after a scale-down?

No. Constraints are only evaluated when a Pod is scheduled. After scale-down the distribution can be uneven; use the Descheduler or a manual rollout restart to rebalance.

Does Kubernetes spread Pods by default without any constraint?

Yes, softly. kube-scheduler applies built-in default constraints (maxSkew: 3 per hostname and maxSkew: 5 per zone, both ScheduleAnyway) to Pods owned by a Service, ReplicaSet, StatefulSet or ReplicationController that define no constraints of their own, provided the nodes carry those labels.

Topology spread or pod anti-affinity for high availability?

Use topology spread when you want controlled balance across domains (several Pods per zone allowed). Use required anti-affinity when you need a strict “no two on the same node” rule. Many production Deployments use a hard zone spread plus a soft hostname spread or soft anti-affinity.

Conclusion

Topology spread constraints look like one field with a number in it, but the scheduler’s behaviour is driven by a precise rule: after placement, no domain may exceed the global minimum by more than maxSkew. Once you can compute that by hand, the rest of the API falls into place. minDomains changes what the global minimum is before all zones exist, matchLabelKeys changes which Pods count during a rollout, the two node-inclusion policies change which nodes form the domains, and the cluster-level defaults explain why an unconstrained Pod is already a little spread out. The limitations, especially no rebalancing after scale-down and the ghost-Pod selector mismatch, are where exam distractors and production incidents both come from.

For the broader picture of how spread fits beside taints, affinity and nodeSelector, go back to the CNCF scheduling guide; for the CKA-wide view of workloads and scheduling, see the CKA workloads & scheduling guide; and for the full path across the five CNCF exams, read the KubeAstronaut certification path guide.