Run kubectl describe node on any worker and you will see two blocks that never match: Capacity and Allocatable. On a kubeadm node with 16 CPUs and 32 GiB, Allocatable may read 15.5 CPU and a little over 31 GiB. On a managed cloud node the gap is far larger. Most CKA candidates know the one-line explanation (“the kubelet reserves resources for itself and the OS”) and stop there, which is fine until a troubleshooting task hands you a Pod stuck Pending with Insufficient cpu on a node that is “obviously” half empty, or a Pod that was scheduled, started, and then went Terminated with a reason you have never seen.
This guide is about that gap and the kubelet machinery around it. The first half covers Node Allocatable: the formula, the KubeletConfiguration fields that set it, what enforceNodeAllocatable actually does to cgroups, and how to read the numbers. The second half covers the three kubelet resource managers that decide which CPUs and which memory a container gets once it has been admitted: the CPU Manager, the Memory Manager and the Topology Manager. All of it sits in the CKA’s Cluster Architecture, Installation & Configuration and Troubleshooting domains, because every piece is a kubelet configuration change on a node you administer. Feature stages below are from the current Kubernetes documentation (v1.37); every feature discussed is GA unless noted.
Capacity, Allocatable and Allocated: Three Numbers That Get Confused
Start by reading a node properly:
kubectl describe node worker-1 | sed -n '/^Capacity:/,/^Events:/p'Capacity:
cpu: 16
ephemeral-storage: 102687672Ki
hugepages-2Mi: 0
memory: 32780152Ki
pods: 110
Allocatable:
cpu: 15500m
ephemeral-storage: 94636830047
hugepages-2Mi: 0
memory: 32063352Ki
pods: 110
...
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 12250m (79%) 14 (90%)
memory 20Gi (65%) 28Gi (90%)| Field | Who sets it | Meaning |
|---|---|---|
| Capacity | kubelet, from the hardware (cAdvisor) | Everything the machine has |
| Allocatable | kubelet, from Capacity minus reservations | What the scheduler may hand to Pods; the scheduler never oversubscribes it |
| Allocated resources | computed by kubectl describe from Pod requests on that node | How much of Allocatable is already promised; percentages are of Allocatable |
The scheduler’s Insufficient cpu message compares a Pod’s requests to Allocatable minus the sum of existing requests, never to live usage and never to Capacity. That single sentence resolves most “but the node is idle” confusion: a node whose containers are idle but request 15.3 of 15.5 allocatable CPUs cannot take a 500m Pod. Our workloads and scheduling guide covers the request/limit side; this guide covers where the 15.5 came from.
The Node Allocatable Formula
Allocatable = Capacity − kubeReserved − systemReserved − evictionHardkubeReserved: resources for Kubernetes system daemons (kubelet, container runtime). Not for daemons that run as Pods; those are ordinary Pods with requests.systemReserved: resources for OS daemons (sshd, udev, journald) and kernel memory, which is not charged to Pods.evictionHard: the hard eviction thresholds. Onlymemoryandephemeral-storageevictions exist, so the eviction term is subtracted from those two resources only. CPU has no eviction threshold and so no eviction term.
Each reservation supports cpu, memory, ephemeral-storage and pid. The documentation’s worked example is worth memorising because its arithmetic is exactly what an exam question can ask you to reproduce:
| Input | Value |
|---|---|
| Capacity | 16 CPU, 32Gi memory, 100Gi ephemeral storage |
kubeReserved | cpu: 1000m, memory: 2Gi, ephemeral-storage: 1Gi |
systemReserved | cpu: 500m, memory: 1Gi, ephemeral-storage: 1Gi |
evictionHard | memory.available: 500Mi, nodefs.available: 10% |
| Allocatable CPU | 16 − 1 − 0.5 = 14.5 |
| Allocatable memory | 32 − 2 − 1 − 0.5 = 28.5Gi |
| Allocatable storage | 100 − 1 − 1 − 10 = 88Gi |
Consequences: the scheduler keeps total Pod memory requests at or below 28.5Gi and storage at or below 88Gi; the kubelet evicts Pods when Pod memory usage exceeds 28.5Gi or disk exceeds 88Gi (that is the pods enforcement explained next); and if every process on the node burned all the CPU it could, Pods collectively could not exceed 14.5 CPUs.
If kubeReserved and systemReserved are not enforced (the default) and the daemons overshoot their reservations, Pods still get evicted, but only when node-level memory usage crosses 31.5Gi (Capacity minus the 500Mi eviction threshold) or disk crosses 90Gi. In other words the reservations shape what the scheduler promises, and the eviction threshold is the actual safety line at the node level. The eviction ordering (who dies first) is covered in our KCSA denial-of-service guide; here we only need the thresholds.
The Linux defaults for evictionHard, which apply when you set nothing, are:
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"So even a node with no explicit reservations has Allocatable memory 100Mi below Capacity and Allocatable storage 10% below Capacity. That is the small gap you see on a stock kubeadm node.
Configuring Reservations in KubeletConfiguration
Everything lives in the kubelet’s config file, which kubeadm writes to /var/lib/kubelet/config.yaml from the kubelet-config ConfigMap in kube-system. A complete, conservative reservation block looks like this:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
kubeReserved:
cpu: "500m"
memory: "1Gi"
ephemeral-storage: "1Gi"
pid: "1000"
systemReserved:
cpu: "500m"
memory: "1Gi"
ephemeral-storage: "1Gi"
pid: "1000"
evictionHard:
memory.available: "200Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
enforceNodeAllocatable:
- podsApply it the way you would in the exam:
sudo vim /var/lib/kubelet/config.yaml # edit the fields above
sudo systemctl restart kubelet
kubectl get node worker-1 -o jsonpath='{.status.allocatable}{"\n"}'Allocatable updates on the next node status report. The pid reservation is worth noticing: it reduces the number of process IDs available to Pods, which is how you leave headroom for the kubelet and runtime during a fork storm (the KCSA guide’s PID-exhaustion section explains the attack this defends against).
A quick way to confirm what the running kubelet believes, without reading files on the node:
kubectl get --raw "/api/v1/nodes/worker-1/proxy/configz" | python3 -m json.tool | grep -A4 -E 'kubeReserved|systemReserved|evictionHard|enforceNodeAllocatable'What enforceNodeAllocatable Actually Does
The reservations above change a number (Allocatable). Enforcement is separate: it is the kubelet applying cgroup limits so that groups of processes cannot exceed their share. enforceNodeAllocatable is a list, and its valid values are:
| Value | What the kubelet enforces | Requires |
|---|---|---|
pods (default) | All Pods together may not exceed Allocatable. Implemented by setting limits on the top-level Pod cgroup (kubepods.slice under systemd). When total Pod usage exceeds Allocatable, Pods are evicted. | cgroupsPerQOS: true (default) |
kube-reserved | Kubernetes daemons may not exceed kubeReserved | kubeReservedCgroup set to the cgroup that contains the kubelet and runtime |
system-reserved | OS daemons may not exceed systemReserved | systemReservedCgroup set |
kube-reserved-compressible | Only the compressible resources (CPU) of kubeReserved | kubeReservedCgroup |
system-reserved-compressible | Only the compressible resources of systemReserved | systemReservedCgroup |
enforceNodeAllocatable: [pods, kube-reserved-compressible, system-reserved-compressible]
kubeReservedCgroup: /runtime.slice # kubelet + containerd live here
systemReservedCgroup: /system.slice # everything systemd startedThree facts about this that catch people:
- The kubelet does not create
kubeReservedCgrouporsystemReservedCgroup. They must already exist, and with thesystemddriver the name must follow the systemd.sliceconvention. If a configured cgroup is missing or invalid, the kubelet fails to start. That is a plausible exam troubleshooting scenario: kubelet down,journalctl -u kubeletcomplaining about a cgroup, fix the config and restart. - Enforcing
system-reservedis dangerous. Memory is incompressible. If sshd, the kernel and journald collectively exceedsystemReserved.memory, something insystem.slicegets OOM-killed, and it might be sshd. The documentation says to do it only after exhaustive profiling, and the-compressiblevariants exist so you can cap CPU (which merely throttles) without risking OOM kills. - The recommended rollout order is: enforce
pods(default); once you have monitoring on daemon usage, add the-compressiblevariants; then non-compressiblekube-reservedbased on observed usage; and only if truly necessary, non-compressiblesystem-reserved.
For the exam the realistic task is the first row: Allocatable is computed from reservations, pods enforcement is on by default, and the two daemon enforcements are opt-in and need pre-existing cgroups.
reservedSystemCPUs: Reserving Specific CPUs
kubeReserved.cpu and systemReserved.cpu reserve an amount of CPU. reservedSystemCPUs reserves an explicit set:
reservedSystemCPUs: "0-1"When set, it overrides the CPU values of kubeReserved and systemReserved for the purpose of deciding which CPUs are kept out of the Pod pool, and it is the mechanism meant for latency-sensitive (telco/NFV) nodes where you want OS daemons and interrupts pinned to known cores. Note that Kubernetes only keeps Pods off those CPUs; actually moving daemons and IRQs onto them needs OS tooling such as tuned or systemd’s CPUAffinity. This option is what makes the CPU Manager’s static policy predictable, which is where the second half of this guide begins.
The CPU Manager
By default, Linux’s Completely Fair Scheduler shares CPUs among all containers: requests become CFS shares (relative weight) and limits become CFS quota (a hard cap per period). That is fine for web services and terrible for workloads sensitive to cache misses, context switches and scheduler jitter. The kubelet’s CPU Manager (GA since v1.26) addresses this with two policies.
cpuManagerPolicy | Behaviour |
|---|---|
none (default) | Every container uses the shared pool; CFS shares and quota do the work |
static | Containers in Guaranteed Pods that request an integer number of CPUs get exclusive CPUs (a dedicated cpuset). Everyone else uses the shared pool |
Who gets exclusive CPUs
The qualification has two parts and both are tested:
- The Pod must be Guaranteed QoS: every container has requests equal to limits for both CPU and memory (or only limits, which Kubernetes copies to requests).
- The container must request an integer number of CPUs (
cpu: "2", not"1500m").
A Guaranteed container with cpu: 1500m runs on the shared pool. A Burstable container with cpu: "4" runs on the shared pool. Only Guaranteed-and-integer containers are pinned.
apiVersion: v1
kind: Pod
metadata:
name: pinned
spec:
containers:
- name: dpdk
image: registry.k8s.io/pause:3.10
resources:
requests: { cpu: "2", memory: "1Gi" }
limits: { cpu: "2", memory: "1Gi" }The shared pool is every CPU not exclusively assigned to a container. Reserved CPUs (from reservedSystemCPUs, or the kubeReserved/systemReserved CPU amounts rounded up) are taken from the shared pool first, in ascending physical core order, and are never handed out exclusively. The static policy therefore requires a non-zero CPU reservation; without one the shared pool could be emptied by pinned containers and the kubelet refuses the configuration. With the strict-cpu-reservation option (GA in v1.35) reserved CPUs are removed from the shared pool entirely, so even Burstable Pods never run on them; without it, the shared pool includes reserved CPUs for non-exclusive workloads.
Enabling it, and the state-file trap
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 10s # default follows nodeStatusUpdateFrequency
kubeReserved:
cpu: "1" # must be > 0 for static
cpuManagerPolicyOptions:
full-pcpus-only: "true"If you simply change the policy and restart the kubelet, it crashloops with:
could not restore state from checkpoint: configured policy "static" differs from
state checkpoint policy "none", please drain this node and delete the CPU manager
checkpoint file "/var/lib/kubelet/cpu_manager_state" before restarting KubeletThe correct procedure, which is a textbook exam task, is:
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
sudo systemctl stop kubelet
sudo rm /var/lib/kubelet/cpu_manager_state
sudo vim /var/lib/kubelet/config.yaml # set cpuManagerPolicy: static (+ reservation)
sudo systemctl start kubelet
kubectl uncordon worker-1The same drain-delete-restart dance is needed whenever the set of online CPUs changes, and when you add strict-cpu-reservation. Our node maintenance guide covers the drain itself.
Static policy options
Options are passed as cpuManagerPolicyOptions (or --cpu-manager-policy-options) and gated by stage: beta options are visible by default (CPUManagerPolicyBetaOptions), alpha options need CPUManagerPolicyAlphaOptions.
| Option | Stage (version) | Effect |
|---|---|---|
full-pcpus-only | GA (1.33) | Allocate whole physical cores, never a lone hyperthread sibling; prevents a neighbour’s thread from sharing your core |
distribute-cpus-across-numa | Beta (1.33) | Spread an allocation evenly across NUMA nodes when it cannot fit in one |
strict-cpu-reservation | GA (1.35) | Keep reserved CPUs out of the shared pool entirely |
prefer-align-cpus-by-uncorecache | GA (1.36) | Prefer CPUs that share an L3/uncore cache |
align-by-socket | Alpha (1.25) | Align allocations to socket boundaries |
distribute-cpus-across-cores | Alpha (1.31) | Spread across physical cores; incompatible with full-pcpus-only and distribute-cpus-across-numa |
You will not be asked to tune these on the CKA, but full-pcpus-only is the one to recognise: it is the standard answer to “a latency-sensitive workload must not share a physical core with another container”.
Verifying pinning
# which CPUs did the container get?
kubectl exec pinned -- cat /sys/fs/cgroup/cpuset.cpus.effective
# kubelet's view of the assignments
sudo cat /var/lib/kubelet/cpu_manager_state | python3 -m json.toolOne consequence that links to a CKAD topic: Pods whose containers hold exclusive CPUs (or Memory Manager reservations) cannot be resized in place; the in-place Pod resize guide explains why.
The Memory Manager
Pinning CPUs is only half of NUMA locality: a container pinned to CPUs on socket 0 whose memory is allocated on socket 1 still pays a cross-socket penalty on every access. The Memory Manager (GA since v1.32) guarantees that Guaranteed Pods receive memory (and hugepages) from the fewest possible NUMA nodes, and it tells the Topology Manager which nodes those are.
memoryManagerPolicy | Behaviour |
|---|---|
None (default) | No effect on placement; returns a neutral topology hint |
Static (Linux) | For Guaranteed Pods, reserves memory in its NUMA map and returns hints for the nodes where the memory can be guaranteed; enforces cpuset.mems on the Pod’s cgroup. BestEffort and Burstable Pods get the default hint and no reservation |
The Static policy has a prerequisite that trips people up: you must tell the Memory Manager how the node’s reserved memory is spread across NUMA nodes with reservedMemory, and the totals must satisfy an exact equation:
Σ reservedMemory[i].memory = kubeReserved.memory + systemReserved.memory + evictionHard.memory.availableBecause evictionHard.memory.available defaults to 100Mi, the sum is never simply the two reservations. The documentation’s example makes the arithmetic explicit:
memoryManagerPolicy: Static
kubeReserved: { cpu: "4", memory: "4Gi" }
systemReserved: { cpu: "1", memory: "1Gi" }
reservedMemory:
- numaNode: 0
limits:
memory: "3Gi"
- numaNode: 1
limits:
memory: "2148Mi" # 3Gi minus 100Mi, so the total is 4Gi + 1Gi + 100MiIf the equation does not balance, the kubelet logs an error at startup and the Memory Manager does not start. Hugepages are reserved per NUMA node the same way (hugepages-1Gi: "2Gi"), and the listed sizes must exist on the machine. The Memory Manager keeps its own checkpoint at /var/lib/kubelet/memory_manager_state, with the same drain-and-delete requirement as the CPU Manager when you change policy.
The Topology Manager
CPU Manager and Memory Manager (and the Device Manager, for GPUs and SR-IOV NICs) each make allocation decisions. Without coordination they can pick different NUMA nodes for the same container. The Topology Manager (GA since v1.27) is the kubelet component that collects hints from those hint providers (each hint is a bitmask of NUMA nodes plus a preferred flag), picks one according to a policy, and admits or rejects the Pod.
Scope
topologyManagerScope | Alignment unit |
|---|---|
container (default) | Each container is aligned independently; two containers in one Pod may land on different NUMA nodes |
pod | All containers in the Pod are treated as one unit and placed on a single NUMA node or a common minimal set; the Pod’s effective request is the max of the app-container sum and the largest init container |
Policies
topologyManagerPolicy | Admits when | On failure |
|---|---|---|
none (default) | Always; no alignment | n/a |
best-effort | Always; records the preferred affinity and tries to honour it | Pod runs anyway |
restricted | Only if a preferred (optimal) affinity exists | Pod rejected at admission |
single-numa-node | Only if every requested resource fits on one NUMA node | Pod rejected at admission |
The failure mode is the troubleshooting fact to burn in. A Pod rejected by restricted or single-numa-node has already been scheduled to the node; the scheduler is not topology-aware. The kubelet refuses it at admission, the Pod goes to phase Failed and shows as Terminated with reason TopologyAffinityError, and the scheduler does not reschedule it, because from its point of view the Pod was placed successfully. A bare Pod simply stays dead; a Deployment or ReplicaSet creates a replacement, which may land on the same node and fail the same way. The documentation recommends a controller (Deployment/ReplicaSet) or an external loop that redeploys Pods with that error.
kubectl get pod pinned -o jsonpath='{.status.phase} {.status.reason} {.status.message}{"\n"}'
# Failed TopologyAffinityError Resources cannot be allocated with Topology localityWhich Pods it considers
Topology Manager asks hint providers about every Pod, but what they return depends on QoS and the request shape: a Guaranteed container with integer CPUs gets real CPU hints from the static CPU Manager; a Guaranteed container with 300m gets the default hint for CPU but still gets device hints if it requests a GPU; BestEffort and Burstable containers get default CPU hints. So alignment effectively applies to Guaranteed integer-CPU workloads and to anything requesting devices.
Options and limits
prefer-closest-numa-nodes(GA in 1.32): withbest-effortorrestricted, prefer NUMA node sets that are closer together (lower NUMA distance) when more than one node is needed.max-allowable-numa-nodes(GA in 1.35): the Topology Manager refuses to run on nodes with more than 8 NUMA nodes by default, because enumerating affinities explodes combinatorially; this raises the cap at your own risk.
A sensible full configuration for a NUMA-sensitive worker, then, is:
cpuManagerPolicy: static
cpuManagerPolicyOptions:
full-pcpus-only: "true"
memoryManagerPolicy: Static
reservedMemory: [...] # balanced per the formula above
topologyManagerPolicy: single-numa-node
topologyManagerScope: pod
reservedSystemCPUs: "0-1"Putting It Together: Admission Order on a Node
When the scheduler binds a Pod to a node, the kubelet runs its own admission before anything starts:
- Fit check against Allocatable: do the Pod’s requests fit in Allocatable minus already-admitted Pods? (Usually yes, because the scheduler used the same numbers, but static Pods and races can fail here with
OutOfcpu/OutOfmemory.) - Topology Manager: collect hints from CPU Manager, Memory Manager and Device Manager; apply the policy; reject with
TopologyAffinityErrorif required. - CPU Manager: assign exclusive CPUs (static policy, Guaranteed integer requests) or leave the container in the shared pool.
- Memory Manager: pin
cpuset.memsfor Guaranteed Pods under the Static policy. - cgroup setup: the Pod goes under
kubepods.slice(and its QoS sub-slice), whose limits implementpodsenforcement of Allocatable.
Only then does the container runtime pull images and start containers. Nothing in steps 2 to 4 is visible to the scheduler, which is why a Pod can be “scheduled” and still never run.
Troubleshooting Table
| Symptom | Likely cause | Check / fix |
|---|---|---|
Pod Pending, Insufficient cpu, node looks idle | Requests exceed Allocatable − allocated requests; usage is irrelevant | kubectl describe node → compare Allocated resources to Allocatable; right-size requests or add nodes |
| Allocatable far below Capacity | Large kubeReserved/systemReserved/evictionHard | Inspect /var/lib/kubelet/config.yaml or the configz endpoint |
Kubelet crashloops after enabling static CPU policy | Stale cpu_manager_state | Drain, stop kubelet, delete the file, start, uncordon |
Kubelet fails to start after adding kube-reserved enforcement | kubeReservedCgroup missing or not a .slice name | journalctl -u kubelet; create the cgroup or fix the name |
| Kubelet logs a Memory Manager error at startup | reservedMemory totals ≠ kube + system reserved + eviction (100Mi default) | Rebalance the per-NUMA values |
Pod Failed / Terminated, reason TopologyAffinityError | restricted or single-numa-node could not satisfy affinity; scheduler will not retry | Reduce the request to fit one NUMA node, relax the policy, or run under a Deployment |
Guaranteed Pod with cpu: 1500m not getting exclusive CPUs | Only integer CPU requests are pinned | Request cpu: "2" (or accept the shared pool) |
Pod OutOfcpu / OutOfmemory at admission | Kubelet’s fit check lost a race or static Pods consumed the headroom | Check static Pod manifests in /etc/kubernetes/manifests; see the static Pods guide |
| Pod evicted although its own usage is modest | pods enforcement: total Pod usage crossed Allocatable, and this Pod ranked first for eviction | Eviction ordering in the KCSA DoS guide; raise requests to Guaranteed where it matters |
| In-place resize rejected on a pinned Pod | Static CPU/Memory manager assignments are fixed at admission | Recreate the Pod; see the CKAD resize guide |
Common Mistakes
- Reading Capacity instead of Allocatable. Every scheduling calculation uses Allocatable. Capacity is the hardware.
- Thinking eviction thresholds reduce CPU Allocatable. Only memory and ephemeral storage have eviction thresholds; CPU Allocatable is Capacity minus the two reservations.
- Assuming reservations are enforced. By default only
podsis enforced. Daemon enforcement is opt-in and needs pre-existing cgroups. - Enabling the static CPU policy without a reservation. The kubelet requires
kubeReserved.cpu,systemReserved.cpuorreservedSystemCPUsto be non-zero. - Changing CPU or Memory Manager policy without deleting the checkpoint file. Drain, delete
/var/lib/kubelet/cpu_manager_state(ormemory_manager_state), restart. - Expecting the scheduler to retry a
TopologyAffinityError. It never does; only a controller recreating the Pod gets another attempt. - Forgetting the 100Mi default eviction threshold in the Memory Manager equation. The reservedMemory sum must include it.
- Confusing
reservedSystemCPUswithkubeReserved.cpu. One names specific CPU IDs to keep Pods away from; the other subtracts an amount from Allocatable. The former overrides the latter’s CPU value when both are set.
How to Practise for the CKA
Set up a two-worker kubeadm cluster (our kubeadm installation guide walks through it) and run these drills against a clock:
- Record
CapacityandAllocatablefor a worker. AddkubeReserved: {cpu: "1", memory: "1Gi"}to/var/lib/kubelet/config.yaml, restart the kubelet, and verify Allocatable dropped by exactly those amounts plus nothing else for CPU and plus the eviction threshold for memory. - Create ten Pods each requesting
cpu: "1"and watch the last one goPendingwithInsufficient cpuwhilekubectl top nodeshows the node nearly idle. Explain the gap in one sentence. - Enable
cpuManagerPolicy: staticthe wrong way (no drain, no file deletion), read the crashloop message injournalctl -u kubelet, then fix it properly. - Deploy a Guaranteed Pod with
cpu: "2"and one withcpu: "1500m"; readcpuset.cpus.effectivein each and confirm only the first is pinned. - Break the Memory Manager equation on purpose, read the startup error, and rebalance
reservedMemory.
These tasks use nothing beyond vim, systemctl, journalctl and kubectl, which is exactly the toolset the exam gives you. If you want the same drills with a timer and a grader, Sailor.sh’s Certified Kubernetes Administrator (CKA) Mock Exam Bundle runs kubelet-configuration and node-troubleshooting scenarios inside real multi-node clusters under the same remote-desktop constraints as the live exam, so you learn where your time actually goes. The CKA troubleshooting guide and the node failure lifecycle guide are the natural companions; the free CKA practice questions are a good warm-up.
Frequently Asked Questions
Why is Allocatable less than Capacity on a Kubernetes node?
Because the kubelet subtracts kubeReserved (for the kubelet and runtime), systemReserved (for OS daemons and the kernel) and, for memory and ephemeral storage, the hard eviction threshold. The scheduler only ever hands out Allocatable.
Does the scheduler look at actual CPU usage when it says Insufficient cpu?
No. It compares the Pod’s requests with Allocatable minus the requests of Pods already on the node. A node can be idle and still full.
What does enforceNodeAllocatable: [pods] do?
It makes the kubelet cap all Pods together at Allocatable through the top-level Pod cgroup and evict Pods when their combined usage exceeds it. It is the default. kube-reserved and system-reserved enforcement are separate opt-ins that require pre-existing cgroups.
Which Pods get exclusive CPUs under the static CPU Manager policy?
Containers in Guaranteed QoS Pods that request an integer number of CPUs. Fractional requests and Burstable or BestEffort Pods stay in the shared pool.
Why does the kubelet crashloop after I set cpuManagerPolicy: static?
The checkpoint file /var/lib/kubelet/cpu_manager_state still records the old policy. Drain the node, stop the kubelet, delete the file, change the config and start the kubelet again.
What is TopologyAffinityError?
The admission failure the kubelet records when the Topology Manager’s restricted or single-numa-node policy cannot satisfy a Pod’s NUMA affinity. The Pod ends up Failed/Terminated and the scheduler does not retry it; a controller must create a new Pod.
Do I need the Memory Manager to use the Topology Manager?
Only for memory and hugepage alignment. CPU alignment needs the CPU Manager static policy; device alignment needs device plugins. Topology Manager coordinates whichever hint providers are active.
Is any of this on the CKA exam?
Node Allocatable, kubectl describe node reading, kubelet configuration edits and restarts, and node troubleshooting are squarely in the Cluster Architecture, Installation & Configuration and Troubleshooting domains. The resource managers appear as kubelet configuration and as the explanation for otherwise puzzling Pod states; you are unlikely to be asked to tune NUMA policy options, but you should recognise the symptoms they cause.
Conclusion
The space between Capacity and Allocatable is where the kubelet keeps the node alive: reservations for its own daemons and the OS, an eviction threshold as the last line of defence, and cgroup enforcement that, by default, confines Pods but not daemons. Learn the formula, know the defaults, and read kubectl describe node from the Allocatable block downward, and the “idle node refuses my Pod” class of question becomes routine.
Below Allocatable sit three managers that decide which CPUs and memory a container gets. The CPU Manager pins Guaranteed integer-CPU containers, the Memory Manager keeps their memory on the same NUMA node, and the Topology Manager makes sure the two agree, rejecting Pods the scheduler cannot know are unplaceable. Their failure signatures (a kubelet crashloop over a stale checkpoint, a Memory Manager equation that will not balance, a TopologyAffinityError the scheduler never retries) are all diagnosable in two commands once you have seen them, which is the whole point of seeing them before exam day.