Every other control-plane component on a kubeadm cluster is configured the same way: edit a static Pod manifest under /etc/kubernetes/manifests, wait for the kubelet to restart it. The kubelet is the exception. It is a systemd service, it reads a configuration file rather than a manifest, that file is generated from a ConfigMap you may never have looked at, and a handful of flags are still layered on top from two different environment files. When a CKA task says “the kubelet on node01 is not starting” or “configure the node to allow 200 Pods”, you are being tested on whether you understand that chain.
Our kubeadm installation guide covers init and join, the troubleshooting guide covers the general method for a broken node, and the Node Allocatable guide covers the reservation fields. This article is about the configuration mechanism itself: where every kubelet setting comes from, in what order they override each other, how to change one safely on one node or on all of them, which defaults the exam expects you to know, and how to read what the kubelet is actually running with. Everything below is based on the Kubernetes v1.37 documentation.
The Startup Chain on a kubeadm Node
Start with what systemd runs. The kubeadm package installs a drop-in unit at /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf that augments the basic kubelet.service. The kubeadm CLI never touches that file; the package owns it. Its content on a current cluster is:
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGSFour variables, four sources, and the order on the ExecStart line is the order of precedence for flags: KUBELET_EXTRA_ARGS is last and therefore wins any conflict. Map each variable to its file and you have the whole picture:
| File | Owner | What it holds |
|---|---|---|
/etc/kubernetes/bootstrap-kubelet.conf | kubeadm, temporary | CA certificate plus bootstrap token used for TLS bootstrap; used only while kubelet.conf does not exist and deleted by kubeadm afterwards |
/etc/kubernetes/kubelet.conf | kubeadm, then the kubelet | the kubelet’s own kubeconfig with its unique client identity, obtained by TLS bootstrap (see the certificate management guide for rotation) |
/var/lib/kubelet/config.yaml | kubeadm | the KubeletConfiguration object, written at init/join and overwritten on every kubeadm upgrade from the cluster-wide ConfigMap |
/var/lib/kubelet/kubeadm-flags.env | kubeadm | one line, KUBELET_KUBEADM_ARGS="...", holding the dynamic, node-specific flags kubeadm decided on at init/join time (the container runtime endpoint, the pause image, --hostname-override or --node-ip if you asked for them) |
/var/lib/kubelet/instance-config.yaml | kubeadm | a local record of the CRI socket kubeadm detected on this node |
/etc/default/kubelet (DEB) or /etc/sysconfig/kubelet (RPM) | you | KUBELET_EXTRA_ARGS="...", the documented last resort for flag overrides; highest priority in the flag chain |
The documentation adds one more layer you should know exists even if you never use it: if you want to override the unit itself, create /etc/systemd/system/kubelet.service.d/ (not the /usr/lib path) and put a file such as local-overrides.conf there. That is standard systemd behaviour, and it is what you would do to add a --config-dir flag permanently, as the drop-in section below shows.
The two commands that show you this chain live, and which you should run before touching anything on a broken node:
systemctl cat kubelet # the unit plus every drop-in, concatenated
ps -ef | grep '[k]ubelet' # the exact flags the running (or last attempted) process gotPrecedence: Config File, Drop-ins and Flags
The kubelet documents an exact merge order at startup, from lowest to highest precedence:
- Feature gates specified on the command line (lowest).
- The kubelet configuration file (
--config). - Drop-in configuration files from
--config-dir, in sort order. - Command-line arguments other than feature gates (highest).
Two things in that list surprise people. First, a flag beats the file: the documentation says plainly that “command line flags which target the same value as a config file will override that value”, which exists for backwards compatibility and is why a stray flag in kubeadm-flags.env can silently defeat an edit you made in config.yaml. Second, --feature-gates on the command line is at the bottom, not the top, so a featureGates: map in the config file or a drop-in wins over the flag.
Almost every kubelet flag that has a config-file equivalent is deprecated as a flag, and the config file is the recommended mechanism because “it simplifies node deployment and configuration management”. The file is a JSON or YAML serialisation of KubeletConfiguration and must carry type metadata:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: "192.168.0.8"
port: 20250
serializeImagePulls: false
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
imagefs.inodesFree: "5%"Relative paths inside the file resolve relative to the file’s own location, while relative paths given as flags resolve relative to the kubelet’s working directory, which is one more reason to keep everything absolute.
Flag defaults and file defaults are not the same
The documentation warns that “some default values differ between command-line flags and the kubelet config file”. The differences are exactly the security-relevant ones, and they are a reliable exam distractor because an old blog post will quote the flag default while your cluster runs the file default:
| Setting | Flag default | KubeletConfiguration default |
|---|---|---|
| anonymous requests to the kubelet API | --anonymous-auth=true | authentication.anonymous.enabled: false |
| kubelet API authorization | --authorization-mode=AlwaysAllow | authorization.mode: Webhook |
| unauthenticated read-only port | --read-only-port=10255 | readOnlyPort: 0 (disabled) |
On any cluster started with --config, the file defaults apply unless a flag overrides them. The hardening story around those three fields belongs to the CKS cluster setup guide; for the CKA you need to know which default you are getting and why ps -ef matters.
Where did --pod-manifest-path go?
Into the file, as staticPodPath. The static Pods guide covers the mechanics; the point here is that whenever a task refers to a classic kubelet flag, your first move is to find the corresponding field in /var/lib/kubelet/config.yaml, and your second is to check that no flag in kubeadm-flags.env or /etc/default/kubelet is overriding it.
Reading the Effective Configuration: /configz
Because configuration can be spread across a file, several drop-ins and flags, the kubelet exposes the final, actuated configuration on its /configz endpoint. You reach it through the API server’s node proxy, which needs nodes/proxy permission, so it works with the admin kubeconfig you have on the exam:
kubectl get --raw "/api/v1/nodes/node01/proxy/configz" | python3 -m json.tool | lessThe response is wrapped in a top-level kubeletconfig object and shows every field with its effective value, including the defaults you never set. It is the fastest way to answer “is this node really running with maxPods: 200?” after an edit, and the only way to see what a drop-in merge produced without reasoning through the rules by hand. The documentation’s own example uses kubectl proxy and curl http://127.0.0.1:8001/api/v1/nodes/<node-name>/proxy/configz | jq ., which is the same request.
Pair it with journalctl -u kubelet on the node. The kubelet validates only the final merged structure, so a bad value in a drop-in shows up as a startup error against the merged config, not against the file you edited.
The —config-dir Drop-in Directory
The drop-in directory solves a real operational problem: kubeadm upgrade overwrites /var/lib/kubelet/config.yaml, so any node-local edit you make there is lost. A drop-in directory is a place kubeadm never writes to.
The mechanics, exactly as documented:
- The kubelet does not look in any directory by default; you opt in with a flag such as
--config-dir=/etc/kubernetes/kubelet.conf.d. - Files must end in
.conf, for example99-kubelet-address.conf. Anything else in the directory is ignored. - Files are processed by sorting the entire file name alphanumerically:
00-kubelet.conffirst, then overridden by01-kubelet.conf. - Each file may be a partial configuration, but must be valid and must include
apiVersionandkind. - Validation runs once, on the final merged structure.
- The only patch strategy is replace: every field present in a higher-priority file replaces the same field from a lower one. This is deliberately different from the kubeadm
--patchesmechanism described later. - For Kubernetes v1.28 and v1.29 the flag only worked if the kubelet process also had the environment variable
KUBELET_CONFIG_DROPIN_DIR_ALPHAset to any value. The feature is stable in current releases and the variable is no longer needed.
Because kubeadm does not pass --config-dir, you add it yourself, and the documented place for a persistent flag is KUBELET_EXTRA_ARGS:
# /etc/default/kubelet (Debian/Ubuntu) or /etc/sysconfig/kubelet (RHEL family)
KUBELET_EXTRA_ARGS="--config-dir=/etc/kubernetes/kubelet.conf.d"Then systemctl daemon-reload && systemctl restart kubelet.
How the merge treats each data type
The merge rules are the part worth memorising, because they decide whether your drop-in did what you think:
| Data type | Behaviour | Example |
|---|---|---|
scalar fields (address, port, maxPods) | later file replaces the value | address: 192.168.0.8 in the drop-in wins |
nested structs (authorization.webhook.*) | merged field by field; untouched sub-fields survive | setting authorization.mode alone keeps cacheAuthorizedTTL from the main file |
lists (clusterDNS, tlsCipherSuites, allowedUnsafeSysctls) | the entire list is replaced, never appended | a drop-in clusterDNS: ["10.96.0.2"] discards the main file’s entries |
maps (featureGates, evictionHard) | merged key by key; keys absent from the drop-in survive | featureGates: {MemoryQoS: false} keeps AllAlpha: false from the main file |
map[string][]string (staticPodURLHeader) | key by key, but each key’s list is replaced whole | only the key you mention is rewritten |
The documentation’s worked example makes the struct rule concrete. Main file:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
port: 20250
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: "5m"
cacheUnauthorizedTTL: "30s"
serializeImagePulls: false
address: "192.168.0.1"Drop-in:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authorization:
mode: AlwaysAllow
webhook:
cacheAuthorizedTTL: "8m"
cacheUnauthorizedTTL: "45s"
address: "192.168.0.8"Result: port: 20250 and serializeImagePulls: false survive because the drop-in never mentioned them, while authorization and address take the drop-in values. For lists the rule is blunter: a main file with clusterDNS: ["192.168.0.9", "192.168.0.8"] and a drop-in with clusterDNS: ["192.168.0.2", "192.168.0.3", "192.168.0.5"] ends up with exactly the three drop-in entries.
One trap sits at the intersection of the map rule and eviction defaults. evictionHard is a map, so a drop-in that sets only memory.available keeps the other thresholds from the main file. But if the main file itself set only one threshold, the others are already zero, because of the eviction default behaviour described later. The drop-in merge cannot restore defaults that the main file discarded.
How kubeadm Manages the Kubelet
kubeadm treats the kubelet configuration as cluster state with a per-node copy.
The kubelet-config ConfigMap
kubeadm init marshals the KubeletConfiguration (the defaults, plus anything you passed under a KubeletConfiguration document in your --config file) to /var/lib/kubelet/config.yaml and uploads the same content to a ConfigMap named kubelet-config in kube-system, under the data.kubelet key. kubeadm join performs TLS bootstrap with the bootstrap token, uses the resulting credential to download that ConfigMap, and writes it to the same local path. That is why every node in a kubeadm cluster starts from an identical kubelet configuration, and why the ConfigMap is the right place to change something cluster-wide.
To see the full default structure before you decide what to change:
kubeadm config print init-defaults --component-configs KubeletConfigurationChanging a setting on every node
The documented reconfiguration loop has three steps, and the exam-relevant detail is that the second one is per node:
# 1. Edit the cluster-wide source of truth (once)
kubectl edit cm -n kube-system kubelet-config
# 2. On each node, one at a time, pull the ConfigMap into /var/lib/kubelet/config.yaml
sudo kubeadm upgrade node phase kubelet-config
# 3. Restart the kubelet on that node
sudo systemctl restart kubeletEditing the ConfigMap alone changes nothing on any running kubelet; the file on disk is what the process reads. Conversely, editing /var/lib/kubelet/config.yaml directly works immediately on that node but is reverted the next time anyone runs kubeadm upgrade node phase kubelet-config or a full kubeadm upgrade, because kubeadm “downloads the KubeletConfiguration from the kubelet-config ConfigMap and overwrite the contents of /var/lib/kubelet/config.yaml”. If a task asks for a durable, cluster-wide change, touch the ConfigMap and then roll it out; if it asks for a quick fix on one node and nothing more, a local edit plus restart is acceptable, but say so in your notes.
The same page documents the sibling procedure for the control plane: ClusterConfiguration lives in the kubeadm-config ConfigMap, and changes to apiServer, controllerManager, scheduler or etcd are regenerated into the static Pod manifests with kubeadm init phase control-plane <component> --config <file> (or kubeadm init phase etcd local), one node at a time. Our cluster upgrade guide shows where these phases run during an upgrade.
Instance-specific configuration
Some settings legitimately differ per node, and kubeadm gives you three ways to express that, in descending order of preference:
| Mechanism | Where | Survives kubeadm upgrade? | Notes |
|---|---|---|---|
kubeadm patches with the kubeletconfiguration target | a patches directory given in InitConfiguration, JoinConfiguration or UpgradeConfiguration (patches.directory), or --patches | only if you pass the same directory again during the upgrade | file naming target[suffix][+patchtype].extension, e.g. kubeletconfiguration0+merge.yaml; patch types strategic (default), merge, json; applied as the last step before the config is written to disk |
nodeRegistration.kubeletExtraArgs | InitConfiguration / JoinConfiguration | written into kubeadm-flags.env, which upgrade preserves | flags, not config fields; in the v1beta4 API it is a list of name/value pairs; check the flag is not deprecated first |
KUBELET_EXTRA_ARGS | /etc/default/kubelet or /etc/sysconfig/kubelet | yes, kubeadm never writes it | the documented “last resort”; highest flag priority |
The documentation lists the classic candidates for instance-specific treatment: --resolv-conf (systemd-resolved hosts need /run/systemd/resolve/resolv.conf), --hostname-override, --container-runtime-endpoint, and historically --cgroup-driver. A node-specific --node-ip on a multi-homed machine is the one you are most likely to meet on the exam, because without it the kubelet picks the node’s default IPv4 address and API server to kubelet traffic (kubectl logs, kubectl exec) can end up on the wrong interface.
Cgroup Drivers and cgroup v2
The kubelet and the container runtime both write to the cgroup tree, and the documentation is categorical: “It’s critical that the kubelet and the container runtime use the same cgroup driver and are configured the same.”
cgroupfs vs systemd
| Driver | Who manages cgroups | When to use |
|---|---|---|
cgroupfs | the kubelet and runtime write directly to the cgroup filesystem | the kubelet’s file default, but “not recommended when systemd is the init system”, because that gives the node two cgroup managers and makes it “unstable under resource pressure” |
systemd | systemd is the single cgroup manager; the kubelet and runtime ask it for cgroups | every systemd-based distribution, and required for cgroup v2 |
Three facts keep this straight on the exam:
-
The raw
KubeletConfigurationdefault forcgroupDriveriscgroupfs, but kubeadm defaults it tosystemdwhen you do not set the field, so a kubeadm-generatedconfig.yamlcontainscgroupDriver: systemd. -
containerd does not default to systemd; you set it in
/etc/containerd/config.toml, and the TOML path depends on the containerd major version:# containerd 1.x [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true # containerd 2.x [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options] SystemdCgroup = trueCRI-O “uses the systemd cgroup driver per default”.
-
Since Kubernetes 1.34 the
KubeletCgroupDriverFromCRIfeature is GA: when the runtime supports theRuntimeConfigCRI call, the kubelet “automatically detects the appropriate cgroup driver from the runtime, and ignores thecgroupDriversetting within the kubelet configuration”. Older runtimes, “specifically, containerd 1.y and below”, do not implementRuntimeConfig, so the kubelet falls back to its own setting; the documentation states that in Kubernetes 1.38 that fallback is dropped and old containerd versions will fail with newer kubelets.
So on a modern pairing (kubelet 1.34+ with containerd 2.x or CRI-O) the mismatch class of bug disappears, because the runtime is the single source of truth. On the mixed clusters the CKA environment may give you, the mismatch is still possible, and its signature is a kubelet that starts, registers the node, and then fails to create Pod sandboxes with cgroup errors in journalctl, or Pods stuck in ContainerCreating on one node only.
Changing the driver on a node that has already joined is “a sensitive operation”: Pods created under the old driver may fail to have their sandboxes recreated, and restarting the kubelet does not fix that. The documented recommendation is to replace the node, or reinstall it with automation, rather than flip the driver in place.
cgroup v2 and failCgroupV1
cgroup v2 has been stable since Kubernetes 1.25 and needs a Linux kernel 5.8 or newer, a runtime that supports it (containerd 1.4+, CRI-O 1.20+), and the systemd driver on both sides. Check which version a node runs with:
stat -fc %T /sys/fs/cgroup/ # cgroup2fs = v2, tmpfs = v1cgroup v1 went into maintenance mode in Kubernetes 1.31 and is deprecated since 1.35. The current documentation states that the kubelet “will no longer start on a cgroup v1 node by default”, and the KubeletConfiguration reference lists failCgroupV1 with a default of true. If you are ever handed an old node image that still boots cgroup v1, the only way to run a current kubelet on it is failCgroupV1: false in the kubelet configuration, and the correct long-term answer is a distribution that boots cgroup v2 (Ubuntu 22.04+, Debian 11+, RHEL 9+ and their relatives all do).
The kubelet detects cgroup v2 automatically and needs no configuration for it. What does need attention is anything that reads the cgroup filesystem inside containers; the documentation calls out older Java runtimes and Node.js 18, which misread v2 memory limits.
Swap: failSwapOn and LimitedSwap
For most of Kubernetes’ history the rule was simple: swap on, kubelet off. The field behind that is failSwapOn, default true, and the kubeadm install guide still describes the two responses: either disable swap (sudo swapoff -a now, and remove it from /etc/fstab or the relevant systemd.swap unit so it stays off after reboot), or tolerate it with failSwapOn: false.
Tolerating swap does not give Pods swap. The documentation is explicit that with failSwapOn: false the kubelet directs the runtime to allocate zero swap to workloads by default, although processes outside Kubernetes, including the kubelet itself, can still swap. Workload swap is a separate decision, made with memorySwap.swapBehavior:
swapBehavior | Effect |
|---|---|
NoSwap (default; an empty value behaves the same) | Pods “do not and cannot use swap” |
LimitedSwap | Burstable Pods may use swap, within a computed limit |
NodeSwap graduated to stable in Kubernetes 1.34, and LimitedSwap comes with rules that make good exam questions:
- Only Burstable QoS Pods are eligible; Guaranteed and BestEffort Pods “are prohibited from utilizing swap memory”.
- High-priority Pods are excluded “to ensure the memory they consume always resides in RAM”.
- A Burstable container can opt out by setting its memory request equal to its limit.
- The per-container limit is
(containerMemoryRequest / nodeTotalMemory) × totalPodsSwapAvailable, so swap is shared in proportion to requested memory. - It needs cgroup v2 (the kubelet sets
memory.swap.maxon the container cgroup) and Linux. - Memory-backed volumes (
secretmounts,emptyDirwithmedium: Memory) are mountedtmpfswith thenoswapoption so their contents stay in RAM; on kernels older than 6.3 the kubelet checks for support and logs a warning if it cannot guarantee it.
The project’s own recommendations: encrypt swap space, run control-plane nodes without swap, taint swap-enabled nodes, prefer SSD-backed swap, and remember that the scheduler does not account for swap at all, so it changes nothing about where Pods land. For the exam, the two-step distinction (failSwapOn lets the kubelet start; swapBehavior lets Pods swap) is the whole point.
The Defaults the Exam Expects You to Know
You will not be asked to recite the reference page, but you will be asked to change a field and recognise a symptom caused by a default. These are the ones that matter, with their KubeletConfiguration defaults from the v1beta1 reference:
| Field | Default | Why it matters |
|---|---|---|
staticPodPath | "" (kubeadm sets /etc/kubernetes/manifests) | no static Pods unless set; see the static Pods guide |
maxPods | 110 | the classic “allow more Pods on this node” task; the Pod CIDR size and the CNI must have room for them too |
podPidsLimit | -1 (unlimited) | fork-bomb protection; a positive value is enforced per Pod |
evictionHard | memory.available: 100Mi, nodefs.available: 10%, nodefs.inodesFree: 5%, imagefs.available: 15% | see the trap below; soft thresholds and evictionPressureTransitionPeriod (5m) are covered in the Node Allocatable guide |
mergeDefaultEvictionSettings | false | set true so that unspecified eviction signals keep their defaults instead of becoming zero |
imageGCHighThresholdPercent / imageGCLowThresholdPercent | 85 / 80 | image GC starts above 85% disk usage and deletes least-recently-used images until 80% |
imageMinimumGCAge | 2m | an image must be unused this long before GC may remove it |
imageMaximumGCAge | 0s (disabled) | stable since 1.35; removes images unused for longer than this regardless of disk pressure; the age tracking resets on kubelet restart |
serializeImagePulls | true | one image pull at a time per node; set false for parallel pulls |
maxParallelImagePulls | unset (no limit) | stable since 1.35; caps parallel pulls, must be ≥1 and requires serializeImagePulls: false when ≥2, or the kubelet refuses to start |
registryPullQPS / registryBurst | 5 / 10 | registry pull rate limiting; 0 disables the QPS limit |
containerLogMaxSize / containerLogMaxFiles | 10Mi / 5 | rotation thresholds; kubectl logs returns only the latest file, so 40 MiB of output yields at most 10 MiB |
containerLogMaxWorkers / containerLogMonitorInterval | 1 / 10s | parallel rotation workers and how often the kubelet checks log sizes |
nodeStatusUpdateFrequency / nodeStatusReportFrequency / nodeLeaseDurationSeconds | 10s / 5m / 40 | heartbeat mechanics; the node failure lifecycle guide explains what happens when they stop |
shutdownGracePeriod / shutdownGracePeriodCriticalPods | 0s / 0s | Graceful Node Shutdown is off until you set them; same guide |
cgroupDriver | cgroupfs (kubeadm writes systemd) | see above |
failSwapOn / memorySwap.swapBehavior | true / NoSwap | see above |
failCgroupV1 | true | kubelet refuses to start on cgroup v1 hosts |
rotateCertificates / serverTLSBootstrap | false / false (kubeadm enables client rotation) | certificate management guide |
authentication.anonymous.enabled / authorization.mode / readOnlyPort | false / Webhook / 0 | the hardened file defaults from the table earlier |
clusterDNS / clusterDomain / resolvConf | set by kubeadm (10.96.0.10, cluster.local) | wrong values break in-Pod DNS on that node only; the CoreDNS guide covers the symptoms |
seccompDefault | false | true applies RuntimeDefault to every container without an explicit profile |
crashLoopBackOff.maxContainerRestartPeriod | unset (effective 300s) | beta and on by default since 1.35; accepts 1s–300s and caps the restart backoff per node |
imagePullCredentialsVerificationPolicy | NeverVerifyPreloadedImages | beta since 1.35; controls whether the kubelet re-checks pull credentials for images already on the node |
featureGates | {} | a map, so drop-ins merge it key by key |
The evictionHard trap
The documentation spells this out and the exam loves it: “by changing the default value of only one parameter for evictionHard, the default values of other parameters will not be inherited and will be set to zero.” A config file that says only
evictionHard:
memory.available: "500Mi"has just disabled disk-pressure eviction entirely, because nodefs.available, nodefs.inodesFree and imagefs.available are now zero. Either list every threshold explicitly or set mergeDefaultEvictionSettings: true so the unspecified ones keep their defaults.
Container restart backoff
The default CrashLoopBackOff curve starts at 10s, doubles each restart (10s, 20s, 40s, …), caps at 300s, and resets after a container has run for 10 minutes. Two newer knobs change it: crashLoopBackOff.maxContainerRestartPeriod in the kubelet config lowers the cap per node (a value under 10s also becomes the initial delay), and the alpha, off-by-default ReduceDefaultCrashLoopBackOffDecay feature gate changes the cluster default to start at 1s and cap at 60s. Per-node configuration wins over the gate. If a troubleshooting task’s Pod seems to take forever to retry after you fixed its image, this is why; kubectl delete pod or a rollout restart skips the wait.
A Symptom-to-Fix Table for a Kubelet That Will Not Start
The method is always the same: systemctl status kubelet, then journalctl -u kubelet -xe --no-pager | tail -50, then read the ExecStart line and open whichever file the error points at. The common failures:
Symptom in journalctl / kubectl | Cause | Fix |
|---|---|---|
failed to load kubelet config file ... strict decoding error: unknown field | typo in a config.yaml field name (or a flag-style name used in the file) | correct the field; the reference page is the source of names; restart |
failed to load kubelet config file ... no such file or directory | wrong --config path, usually a tampered 10-kubeadm.conf or a missing /var/lib/kubelet/config.yaml | fix the path, or regenerate the file with kubeadm upgrade node phase kubelet-config; daemon-reload if you edited a unit |
running with swap on is not supported | swap enabled and failSwapOn: true | swapoff -a plus /etc/fstab, or failSwapOn: false if the task says to tolerate swap |
| kubelet exits immediately on an old image, log mentions cgroup v1 | failCgroupV1: true on a cgroup v1 host | boot a cgroup v2 image, or set failCgroupV1: false as a stopgap |
validate service connection: ... container runtime is not running or connect: no such file or directory | containerd/CRI-O stopped, or --container-runtime-endpoint in kubeadm-flags.env points at the wrong socket | start the runtime; compare the socket with crictl info and /var/lib/kubelet/instance-config.yaml |
node NotReady, Pods stuck ContainerCreating with cgroup errors on one node | cgroup driver mismatch between kubelet and runtime | align cgroupDriver and SystemdCgroup; on a joined node, prefer rebuilding it |
x509: certificate has expired / Unauthorized in kubelet logs | kubelet client certificate expired or kubelet.conf broken | the certificate management guide |
kubelet healthy but kubectl logs/exec time out for that node | wrong node address, often a multi-NIC host without --node-ip | add --node-ip via kubeadm-flags.env or KUBELET_EXTRA_ARGS; check kubectl get node -o wide |
| DNS fails only for Pods on one node | node-specific clusterDNS or resolvConf drift | compare /configz between a good and a bad node |
edit to config.yaml has no effect | a flag in kubeadm-flags.env or /etc/default/kubelet overrides it, or you forgot to restart | `ps -ef |
edit to config.yaml disappeared | kubeadm upgrade rewrote it from the ConfigMap | put the change in the kubelet-config ConfigMap, a patch, or a drop-in directory |
Two universal rules: systemctl daemon-reload is needed after editing a unit or an EnvironmentFile referenced by one, and never needed after editing config.yaml; and kubeadm upgrade node phase kubelet-config is both the repair command for a mangled config.yaml and the reason your local edits vanish.
Exam Drills
Work these on a two-node kubeadm cluster. Each is sized like a real CKA task.
- Raise the Pod limit on one worker. Set
maxPods: 150onnode01only, without changing the ConfigMap. Confirm with/configzandkubectl describe node node01 | grep -A1 Allocatable. - Make the change cluster-wide and durable. Edit the
kubelet-configConfigMap to setmaxPods: 150, roll it out to each node withkubeadm upgrade node phase kubelet-configand a restart, one node at a time, draining first if the task says workloads must not be disrupted. - Use a drop-in. Add
--config-dir=/etc/kubernetes/kubelet.conf.dthroughKUBELET_EXTRA_ARGS, create10-pids.confwithpodPidsLimit: 2048andapiVersion/kind, restart, and prove via/configzthatmaxPodsfrom the main file survived andpodPidsLimitchanged. - Break and fix. Introduce a typo (
maxPodz) inconfig.yaml, restart, read the strict-decoding error injournalctl, and fix it. Then point--configat a non-existent path in a unit override and practisesystemctl cat kubeletto find it. - Swap. Enable swap on a worker (
fallocate,mkswap,swapon), watch the kubelet refuse to start, then solve it both ways:swapoff -awith/etc/fstabcleaned, andfailSwapOn: falsewithswapBehavior: LimitedSwap, checkingkubectl top nodes --show-swap.
Common Mistakes
- Editing the ConfigMap and expecting nodes to change. The ConfigMap is the source;
kubeadm upgrade node phase kubelet-configplus a restart is the delivery. - Editing
config.yamland losing it at the next upgrade. Local edits are overwritten; durable node-specific settings go in a patch,kubeadm-flags.env,KUBELET_EXTRA_ARGSor a drop-in directory. - Forgetting that flags beat the file. A
--max-podsflag lingering inkubeadm-flags.envsilently wins overmaxPodsin the file. Always checkps -ef. - Assuming the file defaults match the flag defaults. Anonymous auth, authorization mode and the read-only port differ.
- Setting one
evictionHardthreshold. The rest become zero unlessmergeDefaultEvictionSettings: true. - Appending to a list in a drop-in. Lists are replaced wholesale; include every entry you want to keep.
- Flipping the cgroup driver on a joined node. Recreate the node instead; and remember that with
KubeletCgroupDriverFromCRIthe runtime’s setting is the one that counts. - Confusing
failSwapOn: falsewith enabling swap for Pods. Pods only swap underswapBehavior: LimitedSwap, and only Burstable ones. - Running
daemon-reloadfor aconfig.yamlchange and skipping it for a unit change. It is the other way round. - Restarting the kubelet on all nodes at once. The documented procedure is one node at a time, so workloads can be rescheduled.
Frequently Asked Questions
Where is the kubelet configuration file on a kubeadm cluster?
/var/lib/kubelet/config.yaml, passed to the kubelet through the --config flag in the KUBELET_CONFIG_ARGS variable of /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf. Dynamic flags live in /var/lib/kubelet/kubeadm-flags.env and user overrides in /etc/default/kubelet (DEB) or /etc/sysconfig/kubelet (RPM). Run systemctl cat kubelet to see the whole chain.
What takes precedence, kubelet flags or the config file?
Flags, with one exception. The documented order from lowest to highest is: --feature-gates on the command line, then the config file, then drop-in files in sort order, then every other command-line flag. So a flag in kubeadm-flags.env overrides the file, but a featureGates map in the file overrides the --feature-gates flag.
How do I change a kubelet setting on every node in a kubeadm cluster?
Edit the kubelet-config ConfigMap in kube-system (kubectl edit cm -n kube-system kubelet-config), then on each node run kubeadm upgrade node phase kubelet-config to rewrite /var/lib/kubelet/config.yaml and systemctl restart kubelet. Do it one node at a time. Editing the ConfigMap alone changes nothing on running kubelets.
How do the kubelet drop-in configuration files merge?
Files in --config-dir must end in .conf, include apiVersion and kind, and are applied in alphanumeric order of their full file names, later files replacing fields from earlier ones. Scalars are replaced, nested structs are merged field by field, maps such as featureGates are merged key by key, and lists such as clusterDNS are replaced in their entirety. Validation happens once on the merged result.
Why must the kubelet and container runtime use the same cgroup driver?
Both manage cgroups for the same containers, and two different managers on a systemd host make the node unstable under resource pressure. kubeadm defaults the kubelet to systemd; containerd must be told with SystemdCgroup = true, while CRI-O defaults to systemd. Since Kubernetes 1.34 the GA KubeletCgroupDriverFromCRI feature lets the kubelet read the driver from a runtime that supports the RuntimeConfig CRI call and ignore its own cgroupDriver setting, which removes the mismatch on modern pairings.
Does the kubelet still refuse to start when swap is enabled?
By default yes, because failSwapOn defaults to true. Set it to false to let the kubelet start on a node with swap, which still gives Pods no swap. To let Burstable Pods use swap, additionally set memorySwap.swapBehavior: LimitedSwap; NodeSwap has been stable since Kubernetes 1.34 and requires cgroup v2.
What happens if I set only one evictionHard threshold?
Every other hard eviction threshold becomes zero, which disables disk-pressure eviction. Specify all of them, or set mergeDefaultEvictionSettings: true so the unspecified signals keep their defaults (memory.available: 100Mi, nodefs.available: 10%, nodefs.inodesFree: 5%, imagefs.available: 15%).
How can I see the configuration a kubelet is actually running with?
Query its /configz endpoint through the API server: kubectl get --raw "/api/v1/nodes/<node>/proxy/configz". The response shows the final merged configuration, including defaults and the effect of any drop-ins, which makes it the fastest way to confirm an edit took effect and to compare a healthy node with a broken one.
Conclusion
The kubelet is configured through a chain, not a file: a systemd unit assembles four variables, the --config file carries most settings, optional drop-ins layer node-specific overrides on top, and flags from kubeadm-flags.env and KUBELET_EXTRA_ARGS sit above all of it. kubeadm keeps the file in step with the kubelet-config ConfigMap, which is why cluster-wide changes go through kubectl edit cm plus kubeadm upgrade node phase kubelet-config, and why local edits vanish at upgrade time. Around that mechanism sit a few defaults with outsized consequences: cgroupDriver must match the runtime unless the runtime is telling the kubelet what to use, failCgroupV1 and failSwapOn decide whether the kubelet starts at all, evictionHard zeroes what you do not mention, and serializeImagePulls, containerLogMaxSize and maxPods shape the behaviour of every node. Read systemctl cat kubelet, ps -ef and /configz before you edit anything, and the “kubelet will not start” task becomes a five-minute item rather than a twenty-minute one.
If you want to rehearse these tasks under time pressure, Sailor.sh’s CKA Certification-Ready Mock Exam Bundle includes kubelet and node configuration scenarios in its Cluster Architecture and Troubleshooting sections across five timed mock exams with explanations, so each wrong turn on precedence or on the reconfiguration loop becomes a fixed habit before exam day. Pair it with the CKA exam guide, the 30-day study plan and the free CKA practice questions.
Continue with the Node Allocatable guide for the reservation and eviction fields, the node failure lifecycle guide for heartbeats and graceful shutdown, and the kubeadm installation guide for the init and join flows that write these files in the first place.