Every kubeadm cluster you touch on the CKA exam has already done something remarkable before you log in: each worker joined with nothing but a 24-hour token, and a few seconds later it held a one-year X.509 client certificate issued by the cluster’s own CA, bound to an identity of the form system:node:<name>, with a controller standing by to renew it before it expires. Nobody ran openssl. Nobody approved anything. That is kubelet TLS bootstrapping, and it is the mechanism underneath kubeadm join, underneath a worker going NotReady because a certificate expired, underneath every Pending CSR you will ever see in kubectl get csr, and underneath the question of why kubectl logs works on one cluster and fails with an “unknown authority” error on another.
This guide is about that machinery. The existing CKA certificate management guide owns kubeadm certs check-expiration, kubeadm certs renew and control-plane certificate expiry, and it gives the kubelet certificate eight lines. The cluster access guide owns the human-user CSR workflow. Neither explains what the kubelet does with the CSR API on its own, which signer it addresses, why its renewal is approved automatically while a user’s never is, what the kubelet-client-current.pem symlink actually is, why the kubelet’s serving certificate is a different story from its client certificate, or how to repair a node whose rotation silently failed six months ago. That is this article. Everything below is checked against the Kubernetes v1.37 documentation, which is the version the live CKA runs on.
Where this sits on the CKA
The CKA weights are Cluster Architecture, Installation & Configuration 25%, Troubleshooting 30%, Services & Networking 20%, Workloads & Scheduling 15% and Storage 10%. Kubelet certificates sit squarely in the first two:
- Cluster Architecture asks you to “create and manage Kubernetes clusters using kubeadm” and “manage the lifecycle of Kubernetes clusters”. Joining a node is TLS bootstrapping; the join token’s purpose is to authenticate a CSR.
- Troubleshooting asks you to “troubleshoot clusters and nodes” and “troubleshoot cluster components”. A node that is
NotReadywithx509: certificate has expired or is not yet validin the API server log is one of the canonical node-level failures, and the fix is notkubeadm certs renew.
You will not be asked to write a custom CSR approver on the exam. You will be expected to read kubectl get csr output and know what each column means, to recognise which CSRs approve themselves and which never will, and to recover a node whose kubelet cannot authenticate.
Two certificates, two directions
The kubelet holds two certificates with opposite jobs, and conflating them is the root of most confusion in this area:
| Certificate | Direction | Who verifies it | Default on kubeadm | How it is obtained | Rotates? |
|---|---|---|---|---|---|
| Client certificate | kubelet → kube-apiserver (every API call the kubelet makes) | kube-apiserver, against --client-ca-file | Issued by the cluster CA via a CSR during join | TLS bootstrapping, signer kubernetes.io/kube-apiserver-client-kubelet | Yes, automatically (rotateCertificates: true) |
| Serving certificate | kube-apiserver → kubelet (port 10250: logs, exec, port-forward, metrics) | kube-apiserver only if --kubelet-certificate-authority is set; metrics-server | Self-signed by the kubelet, written to --cert-dir | Self-signed, or a CSR with signer kubernetes.io/kubelet-serving when serverTLSBootstrap: true | Only if serverTLSBootstrap is on, and then the CSR needs manual or custom approval |
Hold those two rows in mind. The client certificate is what TLS bootstrapping is for. The serving certificate is the optional, less automated second act.
Why bootstrapping exists
The documentation lists the five jobs that fall on whoever runs a cluster with client-certificate authentication:
- Create the CA key and certificate.
- Distribute the CA certificate to the control plane nodes.
- Create a key and certificate for each kubelet, with a unique CN per node.
- Sign each kubelet certificate with the CA key.
- Distribute each key and signed certificate to the right node.
Steps 3 to 5 are the ones that scale badly, and they are the ones TLS bootstrapping automates. The idea, introduced with the certificate request and signing API in Kubernetes 1.4, is simple: give a new node a weak, short-lived credential that is good for exactly one thing, creating a CertificateSigningRequest, then let a controller approve and sign that request, and let the kubelet collect the result and write its own kubeconfig. After that the weak credential is never used again.
The bootstrap sequence, file by file
The documentation describes the sequence in fifteen steps. Here it is compressed, with the kubeadm file names you will actually find on an exam node.
- kubelet starts and looks for the file named by
--kubeconfig. On kubeadm this is/etc/kubernetes/kubelet.conf. On a fresh node it does not exist yet. - Falls back to
--bootstrap-kubeconfig, which kubeadm writes as/etc/kubernetes/bootstrap-kubelet.confduringkubeadm join. This is an ordinary kubeconfig whose user credential is a bootstrap token, and whose cluster entry carries the CA certificate the join process discovered and verified. - Authenticates with the token. The API server recognises the token (it is backed by a Secret of type
bootstrap.kubernetes.io/tokeninkube-system, and--enable-bootstrap-token-auth=trueis on) and authenticates the kubelet assystem:bootstrap:<token-id>in the groupsystem:bootstrappersplus any extra groups the token carries. On kubeadm the extra group issystem:bootstrappers:kubeadm:default-node-token. The token format, Secret layout and expiry mechanics are covered in the CKS API server authentication guide; this article does not re-teach them. - Generates a private key locally and creates a CSR object with
spec.signerName: kubernetes.io/kube-apiserver-client-kubelet, subjectCN=system:node:<nodeName>,O=system:nodes, and usages for client authentication. - The CSR is approved, either automatically by the kube-controller-manager’s approving controller (the normal path) or by a human running
kubectl certificate approve. - The CSR is signed by the kube-controller-manager’s signing controller using the CA referenced by
--cluster-signing-cert-fileand--cluster-signing-key-file. The certificate lands instatus.certificate. - The kubelet collects the certificate, writes key and certificate into
--cert-dir(default/var/lib/kubelet/pki), and writes a real kubeconfig to the--kubeconfigpath. From now onkubelet.confis used andbootstrap-kubelet.confis ignored. - Normal operation begins: the kubelet registers its Node object, starts reporting status, and is scheduled Pods.
- Optionally, rotation: when the certificate nears expiry the kubelet repeats steps 4 to 7 with its current certificate as the credential, and reconnects using the new one.
On an exam node you can see the artefacts of every step:
# The two kubeconfigs the kubelet knows about
sudo ls -l /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/kubelet.conf
# How the kubelet was told about them (kubeadm's systemd drop-in)
cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf | grep -E "bootstrap-kubeconfig|kubeconfig"
# What the bootstrapped kubeconfig points at
sudo grep -E "client-certificate|client-key" /etc/kubernetes/kubelet.conf
# client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
# client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
# The certificates themselves
sudo ls -l /var/lib/kubelet/pki/
That kubelet-client-current.pem line is the detail most candidates have never looked at, and it is the key to understanding rotation, so it gets its own section below.
The identity chain: token, bootstrapper, node
TLS bootstrapping is as much about identity as about certificates. Three identities appear in sequence, each more trusted than the last:
| Stage | Username | Groups | What it may do |
|---|---|---|---|
| Joining | system:bootstrap:<token-id> | system:bootstrappers, system:bootstrappers:kubeadm:default-node-token, system:authenticated | Create and read CSRs (ClusterRole system:node-bootstrapper); have kube-apiserver-client-kubelet CSRs auto-approved |
| Running | system:node:<nodeName> | system:nodes, system:authenticated | Everything the Node authorizer allows for that node; renew its own client certificate; request a serving certificate |
| Human admin | whatever your kubeconfig says | for kubeadm’s admin.conf: kubeadm:cluster-admins | Approve or deny any CSR, including the serving CSRs nobody else will |
The second row is the one every other security control in the node path depends on. The Node authorizer only authorises a kubelet whose certificate puts it in group system:nodes with username exactly system:node:<nodeName>, and <nodeName> must match the registered Node name precisely. The NodeRestriction admission plugin uses the same identity to confine the kubelet to its own Node and its own Pods. The signer enforces the subject, so a kubelet cannot bootstrap itself into someone else’s identity. How the Node authorizer sits in the authorization chain is covered in the KCSA authorization chain guide.
The name-match requirement is also an exam trap: by default the kubelet uses the output of hostname as its node name, or --hostname-override, but with a cloud provider configured the provider may dictate the name. A kubelet whose certificate says system:node:worker-1 while it tries to update a Node called worker-1.example.internal is simply Forbidden.
The five built-in signers
Every CSR names a signer in spec.signerName, and the signer name decides which rules apply and whether anything will ever approve it automatically. Kubernetes ships five:
signerName | Purpose | Auto-approved by kube-controller-manager? | Permitted subject | SANs | Permitted usages |
|---|---|---|---|---|---|
kubernetes.io/kube-apiserver-client | Client certs for humans and tools | Never | Any (but CertificateSubjectRestriction admission rejects O=system:masters) | Honoured | Must include client auth; nothing beyond digital signature, key encipherment, client auth |
kubernetes.io/kube-apiserver-client-kubelet | Kubelet client certs | Yes, when RBAC authorises the requester | O exactly ["system:nodes"], CN system:node:<nodeName> | Forbidden (any SAN fails) | digital signature + client auth, optionally key encipherment |
kubernetes.io/kubelet-serving | Kubelet serving certs | Never | O exactly ["system:nodes"], CN system:node:<nodeName> | DNS and IP SANs honoured, at least one required; email and URI SANs forbidden | digital signature + server auth, optionally key encipherment |
kubernetes.io/legacy-unknown | No trust guarantees; third-party distributions | Never | Any | Honoured | Any |
kubernetes.io/kube-apiserver-serving | Verifying the API server’s own serving cert; trust bundle published as a ClusterTrustBundle | Not handled by kube-controller-manager at all | Follows DNS/IP SAN rules | At least one DNS or IP SAN, must point at the API server | server auth; recommended max lifetime 30 days |
Three consequences worth memorising:
- A user CSR (
kube-apiserver-client) will sitPendingforever until an admin approves it. That is by design and it is why the cluster access guide’s workflow includes an explicitkubectl certificate approve. - A kubelet client CSR (
kube-apiserver-client-kubelet) approves itself on a healthy kubeadm cluster. If you see onePendingfor more than a few seconds, something is broken in the control plane, not in the kubelet. - A kubelet serving CSR (
kubelet-serving) is supposed to sitPending. The built-in approver refuses it on purpose, for the security reason explained below.
The legacy-unknown signer cannot even be requested through the stable certificates.k8s.io/v1 API; it exists for compatibility. The kube-apiserver-serving signer is new territory tied to ClusterTrustBundles, which went GA in 1.37, and is awareness-level for the CKA.
The three controllers inside kube-controller-manager
Nothing about bootstrapping lives in the API server beyond authenticating the request and storing the CSR object. The work happens in three controllers that kube-controller-manager --controllers=* enables by default. Their names in the --controllers list are certificatesigningrequest-approving-controller, certificatesigningrequest-signing-controller and certificatesigningrequest-cleaner-controller.
The approving controller
The approver only handles CSRs whose signer is kubernetes.io/kube-apiserver-client-kubelet. For each one it asks the API server a question via a SubjectAccessReview: is the user who created this CSR authorised to perform the virtual action this CSR represents? Two virtual resources exist, and they map to two ClusterRoles that ship with every cluster:
| Situation | Virtual subresource checked | ClusterRole that grants it | Who should hold it |
|---|---|---|---|
| A brand-new node requesting its first certificate, authenticated by a bootstrap token | certificatesigningrequests/nodeclient | system:certificates.k8s.io:certificatesigningrequests:nodeclient | The bootstrappers group |
| An existing node renewing its certificate, authenticated by its current certificate | certificatesigningrequests/selfnodeclient | system:certificates.k8s.io:certificatesigningrequests:selfnodeclient | system:nodes |
The subtle part: selfnodeclient also verifies that the CSR’s subject matches the requester’s own identity. A node in system:nodes may renew the certificate for system:node:itself, not for another node. That is what the “self” means.
Two behaviours of the approver are exam-relevant. First, it never denies. An unauthorised request is simply ignored and stays Pending, which is why a missing RBAC binding looks like “nothing happens” rather than an error. Second, it is signer-specific: it does not touch kube-apiserver-client or kubelet-serving CSRs at all, regardless of RBAC.
The signing controller
The signer waits for the Approved condition and then issues the certificate with a local CA. It needs the CA material from --cluster-signing-cert-file and --cluster-signing-key-file (on kubeadm these point at /etc/kubernetes/pki/ca.crt and ca.key; in external-CA mode kubeadm sets both to empty strings and the controller signs nothing). You can alternatively give it a different CA per signer with --cluster-signing-kubelet-client-cert-file, --cluster-signing-kubelet-serving-cert-file, --cluster-signing-kube-apiserver-client-cert-file and --cluster-signing-legacy-unknown-cert-file (plus the matching -key-file flags). The generic and per-signer flags are mutually exclusive.
Lifetime is decided by --cluster-signing-duration, default 8760h0m0s (one year). Since Kubernetes 1.22 a CSR may also set spec.expirationSeconds, minimum 600 (ten minutes), and the issued lifetime is the minimum of the two. The flag is therefore a cap, not a fixed value: a requester can ask for shorter, never longer. Signing failures are reported only in kube-controller-manager logs, so a CSR that is Approved but never gets a certificate means you should read that Pod’s logs, not the kubelet’s.
The cleaner controller
CSR objects are not meant to accumulate. The garbage collector removes requests that have not changed state for a fixed period:
| CSR state | Deleted after |
|---|---|
| Approved (and issued) | 1 hour |
| Denied | 1 hour |
| Failed | 1 hour |
| Pending | 24 hours |
| Any state | as soon as the issued certificate has expired |
Exam implication: on a cluster that has been up for a day, kubectl get csr on a healthy system usually shows nothing, and that is normal. If a task says “a node joined an hour ago; inspect its CSR”, the object may already be gone. Conversely a kubelet-serving CSR you forgot to approve will vanish after 24 hours, and the kubelet will create a fresh one.
The RBAC that makes it automatic, and the bindings kubeadm creates
TLS bootstrapping is “just RBAC” once you know the three ClusterRoles involved. The documentation’s generic bindings and kubeadm’s actual bindings differ only in the subject group:
| Purpose | ClusterRole | Generic subject (docs) | kubeadm ClusterRoleBinding name | kubeadm subject |
|---|---|---|---|---|
| Let bootstrapping nodes create and read CSRs | system:node-bootstrapper | group system:bootstrappers | kubeadm:kubelet-bootstrap | group system:bootstrappers:kubeadm:default-node-token |
| Auto-approve first-time node client CSRs | system:certificates.k8s.io:certificatesigningrequests:nodeclient | group system:bootstrappers | kubeadm:node-autoapprove-bootstrap | group system:bootstrappers:kubeadm:default-node-token |
| Auto-approve node client renewal CSRs | system:certificates.k8s.io:certificatesigningrequests:selfnodeclient | group system:nodes | kubeadm:node-autoapprove-certificate-rotation | group system:nodes |
Note that kubeadm binds the more specific bootstrappers group, system:bootstrappers:kubeadm:default-node-token, which is the auth-extra-groups value on the tokens kubeadm token create makes. A token created with a different extra group, or a static token file putting a user only in system:bootstrappers, would authenticate fine but would not be auto-approved on a stock kubeadm cluster. You can verify the bindings exist with one command:
kubectl get clusterrolebinding kubeadm:kubelet-bootstrap kubeadm:node-autoapprove-bootstrap kubeadm:node-autoapprove-certificate-rotation -o wide
For completeness, here is the renewal binding as the documentation writes it generically. On kubeadm you would never create it by hand, but recognising its shape on the exam is useful:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
The CSR API also has its own fine-grained RBAC that explains three validating admission plugins you may meet in flag lists:
- To create a CSR you need
create,get,list,watchoncertificatesigningrequests. - To approve one you need
updateon thecertificatesigningrequests/approvalsubresource and the custom verbapproveon resourcesignerswithresourceNameequal to the signer (kubernetes.io/kube-apiserver-client-kubelet) or a wildcard (kubernetes.io/*). TheCertificateApprovaladmission plugin enforces the signer check. - To sign one you need
updateoncertificatesigningrequests/statusand the verbsignonsigners. TheCertificateSigningadmission plugin enforces it. CertificateSubjectRestrictionrejects anykube-apiserver-clientCSR whose organisation issystem:masters, because that group is hard-wired to cluster-admin and bypasses RBAC.
What kubelet-client-current.pem really is
After bootstrapping, /var/lib/kubelet/pki/ on a kubeadm node looks like this:
$ sudo ls -l /var/lib/kubelet/pki/
kubelet-client-2026-11-11-09-14-02.pem
kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-11-11-09-14-02.pem
kubelet.crt
kubelet.key
Three facts about these files unlock the rest of the topic:
kubelet-client-<timestamp>.pemholds both the private key and the certificate in one PEM file. That is whykubelet.confpointsclient-certificateandclient-keyat the same path.kubelet-client-current.pemis a symlink, not a file. Rotation works by writing a new timestamped file and atomically re-pointing the symlink. Becausekubelet.confreferences the symlink rather than embeddingclient-certificate-data, the kubeconfig never needs to change when the certificate does. (Clusters created with kubeadm older than 1.17 embedded the data and had to be edited by hand; modern kubeadm writes the symlink reference from the start.)kubelet.crtandkubelet.keyare the self-signed serving pair, generated because notlsCertFilewas provided. They have nothing to do with the client identity.
You can read the identity the API server sees straight out of the file:
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -issuer -dates -ext extendedKeyUsage
# subject=O = system:nodes, CN = system:node:worker-1
# issuer=CN = kubernetes
# notBefore=... notAfter=... (one year apart by default)
# X509v3 Extended Key Usage: TLS Web Client Authentication
If the notAfter date is in the past, you have found your NotReady node’s problem.
Client certificate rotation: when it fires and what it needs
Rotation is governed by one KubeletConfiguration field:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
rotateCertificates: true
The field’s upstream default is false; kubeadm sets it to true in the kubelet-config ConfigMap it generates, which is why kubeadm nodes rotate without anyone configuring it. The equivalent flag --rotate-certificates is deprecated in favour of the config file, like most kubelet flags (the full precedence story is in the kubelet configuration guide).
When enabled, the kubelet’s certificate manager:
- Watches the current certificate’s lifetime and picks a renewal moment when between 30% and 10% of the lifetime remains, that is, somewhere between 70% and 90% elapsed, randomised so that a fleet of nodes does not all renew in the same second. For a one-year certificate that is roughly month nine to month eleven.
- Generates a new private key (keys rotate with certificates; it does not re-sign the old key).
- Creates a CSR with signer
kubernetes.io/kube-apiserver-client-kubelet, authenticated with its current certificate, so the requester issystem:node:<name>in groupsystem:nodes. - Relies on the
selfnodeclientbinding to be auto-approved, then on the signing controller to issue. - Writes
kubelet-client-<newtimestamp>.pem, moves the symlink, and reconnects its API server connections using the new certificate.
Because step 3 uses the current certificate, rotation only works while that certificate is still valid. A kubelet that was powered off for a year, or whose control plane was down at every renewal attempt, comes back holding an expired certificate and cannot authenticate to request a new one. That is the failure mode the recovery section fixes, and it is the only situation where you do this by hand.
Two kubelet metrics tell you whether rotation is healthy without reading logs: kubelet_certificate_manager_client_ttl_seconds (seconds until the client certificate expires; negative means already expired; +Inf means unused or invalid) and kubelet_certificate_manager_client_expiration_renew_errors (a counter that should stay at zero). The CIS Kubernetes Benchmark checks that --rotate-certificates is not set to false, so turning rotation off is both operationally and compliance-wise wrong.
The serving certificate and serverTLSBootstrap
Now the second certificate. By default the kubelet self-signs a serving certificate for its HTTPS endpoint on port 10250, saved as kubelet.crt and kubelet.key in --cert-dir, because no tlsCertFile and tlsPrivateKeyFile were supplied. Three parties connect to that endpoint: the API server (for kubectl logs, exec, attach, port-forward, cp), metrics-server (for kubectl top), and anything else scraping kubelet metrics.
Here is the part that surprises people: by default the API server does not verify the kubelet’s serving certificate at all. It authenticates itself to the kubelet with --kubelet-client-certificate and --kubelet-client-key, but it only verifies the kubelet’s identity if --kubelet-certificate-authority is set, which kubeadm does not do, precisely because the kubelets’ certificates are self-signed and could not be verified. The CIS Benchmark flags this (the API server check for --kubelet-certificate-authority), and the fix requires giving kubelets real serving certificates first.
That is what serverTLSBootstrap is for:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
rotateCertificates: true
With serverTLSBootstrap: true, instead of self-signing, the kubelet creates a CSR with signer kubernetes.io/kubelet-serving, subject O=system:nodes, CN=system:node:<name>, with its hostname and IP addresses as DNS and IP SANs, and usages for server authentication. The field requires the RotateKubeletServerCertificate feature gate, which has been beta and enabled by default since 1.12, so on any cluster you meet it is on. Enabling the gate makes the kubelet both request the serving certificate after client bootstrapping and rotate it; the deprecated flag equivalent is --rotate-server-certificates.
And then the CSR sits there:
$ kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 <none> Pending
csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 <none> Pending
The core approving controller deliberately refuses to approve kubelet-serving CSRs, for a documented security reason: a kubelet (or anyone holding a kubelet’s client certificate) could request a serving certificate with any DNS name or IP in the SANs, including the API server’s, and nothing in the request itself proves the node owns those addresses. Automatic approval would turn a single compromised node certificate into a certificate-minting facility. So approval must come from a human (kubectl certificate approve csr-9wvgt) or from a custom approver controller that verifies, out of band, that the requested SANs really belong to that node. The documentation’s checklist for any such approver is worth knowing because exam distractors are built from it: approve only CSRs whose spec.username is system:node:<nodeName> and whose spec.groups contains system:nodes, whose usages contain server auth (optionally digital signature and key encipherment) and nothing else, and whose SANs are only IP and DNS names belonging to the requesting node, with no URI or email SANs.
Once approved, the kubelet writes kubelet-server-current.pem (another symlink), and the serving certificate expires after one year like everything else. Because rotateCertificates is true, a new serving CSR appears before expiry and must be approved again, every year, for every node. Forgetting that is how clusters that “worked for a year” suddenly lose kubectl logs. The metric kubelet_certificate_manager_server_ttl_seconds reports the shortest serving-certificate TTL and kubelet_server_expiration_renew_errors counts failures.
To enable this on an existing kubeadm cluster, the documented procedure is: edit the kubelet-config ConfigMap in kube-system to add serverTLSBootstrap: true (so future joins and kubeadm upgrade node pick it up), add the same field to /var/lib/kubelet/config.yaml on each node, systemctl restart kubelet, approve the resulting CSRs, and only then add --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt to the API server manifest. Do it in the other order and kubectl logs breaks cluster-wide with “x509: certificate signed by unknown authority” until every node has a CA-signed serving certificate.
—cluster-signing-duration and short-lived certificates
Everything the built-in signers issue defaults to one year because --cluster-signing-duration defaults to 8760h0m0s. Operators who want shorter kubelet certificates change that flag in /etc/kubernetes/manifests/kube-controller-manager.yaml; the kubelet restarts the static Pod and every certificate issued from then on is shorter. Existing certificates keep their original expiry until they rotate.
Shorter means more CSR churn (each node rotates at 70 to 90% of a shorter window), which is harmless for client certificates because they auto-approve, but becomes a manual approval treadmill for serving certificates unless you run an approver. A requester can also ask for less than the cap with spec.expirationSeconds; the kubelet-serving and kube-apiserver-client-kubelet CSRs the kubelet generates do not set it (that is the <none> in the REQUESTEDDURATION column), so they always receive the full cap.
The kubeadm certs commands never touch any of this. kubeadm certs check-expiration reports control-plane certificates and the .conf kubeconfigs and explicitly leaves kubelet.conf out, “because kubeadm configures kubelet for automatic certificate renewal with rotatable certificates under /var/lib/kubelet/pki”. If an exam task asks you to inspect kubelet certificate expiry, the tool is openssl on the PEM file, not kubeadm.
What about kube-proxy and other node components?
TLS bootstrapping is a kubelet feature. kube-proxy also talks to the API server from every node, yet it never appears in kubectl get csr. The documentation offers two routes for such components: distribute certificates the old way, or run them as a DaemonSet with a ServiceAccount, which is what kubeadm does. kube-proxy runs in kube-system as a DaemonSet and authenticates with a projected ServiceAccount token, so its identity comes from the token machinery (owned by the CKS ServiceAccount token guide), not from certificates. CNI agents, node exporters and CSI node plugins follow the same pattern. The only process on a node that must use the CSR flow is the kubelet, because it exists before any Pod can.
Node identity enforcement: the Node authorizer and NodeRestriction
The bootstrapping signer’s strict subject rules only matter because two control-plane components rely on that subject.
The Node authorizer (--authorization-mode=Node,RBAC, which kubeadm sets, or type: Node in a structured AuthorizationConfiguration) grants a kubelet exactly what it needs: read services, endpoints, nodes and pods; read the Secrets, ConfigMaps, PVCs and PVs referenced by Pods bound to its node; write its own node and node status, pods and pod status, and events; use the CSR API for bootstrapping; and create TokenReviews and SubjectAccessReviews. With the AuthorizeNodeWithSelectors behaviour, kubelets are limited to reading their own Node object and only the Pods bound to them. None of this applies unless the identity is system:node:<name> in system:nodes.
The NodeRestriction admission plugin (--enable-admission-plugins=NodeRestriction, on by default in kubeadm) then confines writes:
- A kubelet may modify only its own Node object and only Pods bound to it, may not delete its Node, and may not add or remove taints.
- Labels under
kubernetes.io/andk8s.io/are policed. Forbidden for kubelets: anything prefixednode-restriction.kubernetes.io/(reserved for administrators to pin workloads to trusted nodes) andnode-role.kubernetes.io/(so a node cannot declare itself a control-plane node). Allowed:kubernetes.io/hostname,kubernetes.io/arch,kubernetes.io/os,node.kubernetes.io/instance-type,topology.kubernetes.io/regionand/zone, and anything underkubelet.kubernetes.io/ornode.kubernetes.io/. Everything else under those prefixes is reserved. - With
ServiceAccountNodeAudienceRestriction, a kubelet may only request ServiceAccount tokens for audiences already referenced by Pods on its node, unless RBAC grants the custom verbrequest-serviceaccounts-token-audience.
The CKA angle: if a task asks you to label a node node-role.kubernetes.io/worker= you do it as an admin with kubectl label, which is fine. If you try to achieve the same by adding --node-labels=node-role.kubernetes.io/worker= to the kubelet’s flags, the kubelet’s own registration is rejected and the node never becomes Ready. Likewise, the error nodes "worker-2" is forbidden: node "worker-1" is not allowed to modify node "worker-2" in a kubelet log means the certificate identity and the node name disagree.
Beyond kubelets: ClusterTrustBundle and PodCertificateRequest (awareness)
Two newer certificates.k8s.io/v1 kinds reached GA in Kubernetes 1.37 and reuse the signer concept. Neither is CKA core, but both appear in kubectl api-resources output on a current cluster and recognising them avoids confusion:
- ClusterTrustBundle is a cluster-scoped object that distributes X.509 trust anchors (CA certificates) to workloads. Signer-linked bundles are named
<signerDomain>:<signerPath>:<name>and can only be written by a controller holding the custom verbatteston that signer; signer-unlinked bundles have no colon in their name and plain RBAC. All ServiceAccounts can read them by default, and Pods consume them through theclusterTrustBundleprojected volume source. The kube-controller-manager’skube-apiserver-serving-clustertrustbundle-publisher-controllerpublishes the bundle for thekubernetes.io/kube-apiserver-servingsigner. (The feature gate was alpha 1.27, beta off by default 1.33, GA 1.37.) - PodCertificateRequest is a CSR-like object the kubelet creates on a Pod’s behalf when the Pod uses a
podCertificateprojected volume. It carriespodName,podUID,serviceAccountName,nodeName, amaxExpirationSeconds(default 24 hours) and a stub PKCS#10 request holding just the public key. Unlike CSRs there is no approval phase: the signer’s controller issues, denies or fails the request directly, writes the chain intostatus.certificateChainplusnotBefore,notAfterandbeginRefreshAt, and the kubelet handles refresh. NodeRestriction ensures a node can only create requests for Pods actually running on it, and a kube-controller-manager cleaner deletes requests older than 15 minutes. (Alpha 1.34, beta 1.35, GA 1.37.)
Exam value: if you see podcertificaterequests in kubectl get csr’s neighbourhood, it is workload identity, not node identity, and it never needs kubectl certificate approve.
Repairing a node whose kubelet certificate expired
This is the one hands-on recovery procedure in the topic, taken from the kubeadm troubleshooting documentation, and it is a plausible Troubleshooting task because the symptoms are so specific:
kubectl get nodesshows the nodeNotReady.journalctl -u kubeleton the node showsUnauthorizedorx509: certificate has expired or is not yet validwhen talking to the API server, and the kube-apiserver log shows the samex509message for that client.openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -datesconfirmsnotAfteris in the past.
Rotation cannot self-heal here because the kubelet’s only credential is the expired certificate. The documented fix re-bootstraps the node with a fresh kubeconfig signed by the CA:
# 1. On the FAILED node: back up and remove the stale kubeconfig and client certs
sudo mkdir -p /root/kubelet-backup
sudo mv /etc/kubernetes/kubelet.conf /root/kubelet-backup/
sudo mv /var/lib/kubelet/pki/kubelet-client* /root/kubelet-backup/
# 2. On a WORKING control-plane node that has /etc/kubernetes/pki/ca.key:
# mint a kubeconfig whose embedded cert is CN=system:node:<NODE>, O=system:nodes
NODE=worker-1
sudo kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE > kubelet.conf
# If the server/cluster name inside is not right for your cluster, fix them,
# or pass --config with the correct ClusterConfiguration.
# 3. Copy kubelet.conf to the failed node as /etc/kubernetes/kubelet.conf
# 4. On the FAILED node: restart and wait for the symlink to be recreated
sudo systemctl restart kubelet
sudo ls -l /var/lib/kubelet/pki/kubelet-client-current.pem
# 5. Point kubelet.conf back at the rotatable symlink: replace
# client-certificate-data / client-key-data with
# client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
# client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
# then
sudo systemctl restart kubelet
# 6. Verify
kubectl get nodes
Why step 5 matters: the kubeconfig kubeadm kubeconfig user emits embeds a certificate. If you leave it that way, the kubelet will happily bootstrap a rotatable certificate into /var/lib/kubelet/pki but keep authenticating with the embedded one, which will expire again in a year with no rotation. Re-pointing the kubeconfig at the symlink restores the self-healing loop. If your control plane has no ca.key (external CA mode), you must sign the embedded certificate externally instead.
What you should not do: kubeadm certs renew all. It renews control-plane certificates and the admin, controller-manager and scheduler kubeconfigs, and does nothing for kubelet.conf or /var/lib/kubelet/pki. Running it is harmless but wastes exam minutes.
Diagnosis table
| Symptom | Where you see it | Likely cause | Fix |
|---|---|---|---|
kubeadm join fails with Unauthorized | join output | Bootstrap token expired (default TTL 24 h) or deleted | kubeadm token create --print-join-command on a control plane node |
kubeadm join fails with certificate signed by unknown authority | join output | Wrong --discovery-token-ca-cert-hash or wrong CA in bootstrap kubeconfig | Recompute the hash from /etc/kubernetes/pki/ca.crt; re-run join |
Node never appears; kubelet log csr ... is Pending for minutes | kubelet log, kubectl get csr | Approver not running, or no nodeclient binding for the token’s group | Check kube-controller-manager Pod and logs; check kubeadm:node-autoapprove-bootstrap binding; as a stopgap kubectl certificate approve |
CSR Approved but no certificate, kubelet keeps waiting | kubectl get csr shows Approved without Issued | Signing controller cannot sign: missing or empty --cluster-signing-*-file, external CA mode | Read kube-controller-manager logs; signing failures appear only there |
kubectl get csr is empty although a node joined an hour ago | kubectl get csr | Cleaner deleted the approved CSR after 1 hour | Normal; nothing to fix |
kubelet-serving CSRs stay Pending | kubectl get csr | By design: never auto-approved | kubectl certificate approve <name> or run a SAN-verifying approver |
kubectl logs and exec fail cluster-wide with x509: certificate signed by unknown authority | kubectl output | --kubelet-certificate-authority set on the API server while kubelets still self-sign | Approve serving CSRs first, or remove the flag until every node has a signed serving cert |
kubectl top nodes shows <unknown> or metrics-server logs TLS errors | metrics-server logs | metrics-server cannot verify self-signed kubelet certificates | Enable serverTLSBootstrap and approve serving CSRs, or configure metrics-server to skip kubelet TLS verification (less secure) |
Node NotReady; API server log x509: certificate has expired or is not yet valid for that client | kube-apiserver log, journalctl -u kubelet | Kubelet client cert expired and rotation could not run | Re-bootstrap with kubeadm kubeconfig user --org system:nodes --client-name system:node:<NODE> as above |
Node NotReady; kubelet log node "X" is not allowed to modify node "Y" | kubelet log | Certificate CN does not match the registered node name (--hostname-override, cloud provider naming) | Make the node name match the certificate, or re-bootstrap with the right --client-name |
Kubelet rejected when registering with --node-labels=node-role.kubernetes.io/... | kubelet log | NodeRestriction forbids kubelets from setting node-role.kubernetes.io/ and node-restriction.kubernetes.io/ labels | Remove the flag; label from an admin kubeconfig |
| A user’s CSR is never approved | kubectl get csr | kube-apiserver-client signer is never auto-approved | Expected; approve manually (cluster access guide) |
| A CSR fails or the signer refuses it with a subject error | kubectl describe csr, kube-controller-manager log | O not exactly system:nodes, CN not system:node:<name>, or a SAN on a kube-apiserver-client-kubelet request | Regenerate the request with the right subject and no SANs |
After kubeadm certs renew all, node still NotReady | kubectl get nodes | Kubelet certs are not managed by kubeadm certs | Follow the kubelet recovery procedure instead |
Hands-on: read the whole chain on a kubeadm cluster in ten minutes
Run this on any kubeadm practice cluster; nothing here is destructive except the optional serving-certificate step, which is reversible.
# 1. The kubelet's two kubeconfigs and how systemd passes them
cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
sudo grep -E "server:|client-certificate|client-key" /etc/kubernetes/kubelet.conf
# 2. The client certificate identity and expiry
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -dates -ext extendedKeyUsage
# 3. The rotation setting kubeadm wrote
grep -E "rotateCertificates|serverTLSBootstrap" /var/lib/kubelet/config.yaml
kubectl -n kube-system get cm kubelet-config -o yaml | grep -E "rotateCertificates|serverTLSBootstrap"
# 4. The RBAC that auto-approves bootstrap and rotation
kubectl get clusterrolebinding kubeadm:kubelet-bootstrap kubeadm:node-autoapprove-bootstrap kubeadm:node-autoapprove-certificate-rotation -o wide
kubectl describe clusterrole system:node-bootstrapper system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
# 5. The signing cap on the controller manager
sudo grep cluster-signing /etc/kubernetes/manifests/kube-controller-manager.yaml
# 6. Does the API server verify kubelet serving certs?
sudo grep -E "kubelet-certificate-authority|kubelet-client-certificate" /etc/kubernetes/manifests/kube-apiserver.yaml
# 7. (Optional, reversible) Request a signed serving cert on ONE node
sudo sed -i 's/^serverTLSBootstrap:.*//' /var/lib/kubelet/config.yaml
echo "serverTLSBootstrap: true" | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
kubectl get csr # a kubernetes.io/kubelet-serving CSR appears, Pending
kubectl certificate approve <csr-name>
sudo ls -l /var/lib/kubelet/pki/ # kubelet-server-current.pem symlink now exists
# 8. Watch a CSR get garbage-collected after an hour, or just note the timings
kubectl get csr -o custom-columns=NAME:.metadata.name,SIGNER:.spec.signerName,REQUESTOR:.spec.username,AGE:.metadata.creationTimestamp
If you want to practise the recovery procedure, do it on a disposable worker: move kubelet.conf and the kubelet-client* files aside, watch the node go NotReady, then follow the six steps above. Doing it once makes the exam version a five-minute task instead of a twenty-minute panic.
Scenario drill
| Scenario | Answer |
|---|---|
A new worker runs kubeadm join. Which signer does its first CSR address? | kubernetes.io/kube-apiserver-client-kubelet |
| Who approves that CSR on a stock kubeadm cluster? | kube-controller-manager’s approving controller, via a SubjectAccessReview against the nodeclient virtual subresource |
| Which group must the joining identity be in for kubeadm’s auto-approve binding to match? | system:bootstrappers:kubeadm:default-node-token |
| A year later the same kubelet renews. Which ClusterRole authorises the auto-approval? | system:certificates.k8s.io:certificatesigningrequests:selfnodeclient, bound to system:nodes |
| When does the kubelet begin renewal? | At a random point when 30% to 10% of the certificate lifetime remains |
kubelet.conf references client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem. What is that file? | A symlink to the latest timestamped PEM containing key and certificate; rotation re-points it |
| Default issued lifetime, and the flag that caps it? | One year; --cluster-signing-duration default 8760h0m0s |
A CSR sets spec.expirationSeconds: 300. Result? | Rejected: minimum is 600 seconds |
A CSR sets spec.expirationSeconds: 86400 with the default cap. Issued lifetime? | 24 hours, the minimum of the request and the cap |
kubectl get csr shows kubernetes.io/kubelet-serving CSRs Pending for 20 minutes. Bug? | No; the built-in approver never approves serving CSRs; approve manually or run a verifying approver |
| Why does the built-in approver refuse serving CSRs? | The request could carry any DNS or IP SAN, so automatic approval would let a compromised node mint certificates for arbitrary names |
| Which KubeletConfiguration field makes the kubelet request a signed serving certificate? | serverTLSBootstrap: true (requires RotateKubeletServerCertificate, beta on by default since 1.12) |
| By default, does kube-apiserver verify the kubelet’s serving certificate? | No; only when --kubelet-certificate-authority is set, which kubeadm does not do |
| How long does an approved CSR stay in the cluster? | 1 hour (denied and failed also 1 hour; pending 24 hours; any CSR is removed once its certificate expires) |
| A kubelet client CSR includes a DNS SAN. Outcome? | Refused by the kube-apiserver-client-kubelet signer, which forbids SANs |
A user CSR requests O=system:masters. Outcome? | Rejected by the CertificateSubjectRestriction admission plugin |
Node NotReady, API server logs x509: certificate has expired, kubelet cert notAfter in the past. Fix? | Re-bootstrap: remove kubelet.conf and kubelet-client*, generate a kubeconfig with kubeadm kubeconfig user --org system:nodes --client-name system:node:<NODE>, restart, then point the kubeconfig at the symlink |
Does kubeadm certs renew all fix that node? | No; kubeadm does not manage kubelet client certificates |
A kubelet starts with --node-labels=node-role.kubernetes.io/infra=. Result? | NodeRestriction rejects the registration; node-role.kubernetes.io/ is forbidden for kubelets |
Kubelet log: node "w1" is not allowed to modify node "w1.internal". Cause? | Certificate CN system:node:w1 does not match the registered node name |
kube-proxy never shows up in kubectl get csr. Why? | It runs as a DaemonSet with a ServiceAccount token, not with a bootstrapped certificate |
| Which object distributes CA trust anchors to Pods in 1.37? | ClusterTrustBundle (GA 1.37), consumed via a clusterTrustBundle projected volume |
Which object does the kubelet create for a Pod using a podCertificate projected volume, and does it need approval? | PodCertificateRequest (GA 1.37); no approval phase, the signer issues or denies directly, and requests are cleaned up after 15 minutes |
Mistakes that cost points
- Treating
kubeadm certs renewas the fix for a kubelet certificate problem. It renews control-plane certificates and three kubeconfigs.kubelet.confis deliberately excluded because the kubelet rotates its own. - Approving every
PendingCSR reflexively. On the exam akube-apiserver-clientCSR for a user should be approved when the task says so; akubelet-servingCSR should be approved after checking the requestor issystem:node:<name>; akube-apiserver-client-kubeletCSR stuck inPendingmeans the control plane is broken and approving it by hand hides the real fault. - Forgetting that rotation needs a valid current certificate. Once it has expired, no amount of restarting the kubelet will renew it.
- Enabling
--kubelet-certificate-authoritybefore kubelets have signed serving certificates, which breakslogsandexeccluster-wide. - Assuming serving certificates rotate silently like client certificates. They rotate, but each rotation produces a CSR that again needs approval.
- Looking for
kubelet.confexpiry inkubeadm certs check-expiration. It is not listed; useopensslonkubelet-client-current.pem. - Setting node-role labels from the kubelet. NodeRestriction forbids it; label from an admin context.
- Reading kubelet logs for a signing problem. Signing failures are reported only in kube-controller-manager logs.
Frequently Asked Questions
Does the kubelet’s client certificate really renew itself with no human involvement?
Yes, on a kubeadm cluster. kubeadm sets rotateCertificates: true, and the renewal CSR is approved automatically because the requester is in group system:nodes and kubeadm’s kubeadm:node-autoapprove-certificate-rotation binding grants the selfnodeclient ClusterRole. The signing controller issues the certificate, the kubelet swaps the kubelet-client-current.pem symlink, and it reconnects. The only prerequisite is that the current certificate is still valid when the kubelet tries, which is why a long-powered-off node needs the manual recovery procedure.
Why does a user’s CSR need kubectl certificate approve but a node’s does not?
Because they address different signers. kubernetes.io/kube-apiserver-client is never auto-approved by design, since the subject can be anything and a wrongly issued certificate could be cluster-admin. kubernetes.io/kube-apiserver-client-kubelet is auto-approved only when a SubjectAccessReview confirms the requester is authorised for nodeclient or selfnodeclient, and the signer enforces that the subject is exactly O=system:nodes, CN=system:node:<name> with no SANs, so the worst a successful request can obtain is a node identity for the requester itself.
Why are kubelet serving CSRs left Pending on every cluster I have seen?
The core approving controller refuses kubernetes.io/kubelet-serving requests for a documented security reason: the CSR can request any DNS or IP SAN, and nothing in the cluster can automatically prove the node owns those addresses. Approval has to come from an administrator or from a custom controller that verifies the SANs out of band. Most kubeadm clusters leave serverTLSBootstrap off and live with self-signed serving certificates, which is why the API server does not verify them by default.
What is the difference between rotateCertificates and serverTLSBootstrap?
rotateCertificates controls renewal of the client certificate the kubelet presents to the API server and is true on kubeadm. serverTLSBootstrap tells the kubelet to obtain its serving certificate for port 10250 from the CSR API instead of self-signing, and is false by default. With both on, the serving certificate also rotates, but every rotation creates a CSR that needs approval.
How long are the issued certificates valid, and can I change it?
One year by default, from --cluster-signing-duration=8760h0m0s on kube-controller-manager. Lower it in /etc/kubernetes/manifests/kube-controller-manager.yaml to shorten every certificate issued after the restart. Requesters may also ask for less with spec.expirationSeconds (minimum 600 seconds); the issued lifetime is the smaller of the two values.
My node is NotReady and the API server says the kubelet’s certificate has expired. Is there a quick fix?
Yes, and it is not kubeadm certs renew. Back up and remove /etc/kubernetes/kubelet.conf and /var/lib/kubelet/pki/kubelet-client*, generate a new kubeconfig on a control plane node with kubeadm kubeconfig user --org system:nodes --client-name system:node:<NODE>, copy it to the node, restart the kubelet, then edit the kubeconfig to reference /var/lib/kubelet/pki/kubelet-client-current.pem for both certificate and key and restart again. The last step restores automatic rotation; skipping it leaves you with an embedded certificate that will expire again.
Are ClusterTrustBundle and PodCertificateRequest on the CKA?
They are not named in the curriculum, but both are GA in Kubernetes 1.37, the version the exam runs on, and both live in the same certificates.k8s.io API group as CSRs. Know what they are: ClusterTrustBundle distributes CA certificates to workloads, PodCertificateRequest provisions workload certificates through a kubelet-managed projected volume with no approval phase. Neither replaces kubelet TLS bootstrapping for node identity.
Conclusion
Kubelet TLS bootstrapping is the mechanism that turns a 24-hour join token into a one-year node identity and keeps that identity alive without an administrator: a bootstrap kubeconfig that is only good for creating a CSR, a signer that only issues system:node:<name> client certificates with no SANs, an approver that says yes only after a SubjectAccessReview against nodeclient or selfnodeclient, a one-year cap from --cluster-signing-duration, a symlink that lets the kubeconfig stay constant while the certificate changes, and a cleaner that tidies the evidence away after an hour. The serving certificate is the deliberate exception: it is self-signed unless you ask for serverTLSBootstrap, and even then every CSR waits for a human or a verifying controller, because automatic approval would be a certificate-minting hole. Once you can read kubectl get csr and /var/lib/kubelet/pki fluently, the scary exam symptom, a NotReady node with an expired certificate, becomes a six-step procedure you have already rehearsed.
If you want to rehearse it under exam conditions, Sailor.sh’s CKA Certification-Ready Mock Exam Bundle gives you real kubeadm clusters in the browser across five timed, performance-based mock exams with explanations, including node-level troubleshooting scenarios where the certificate chain is the thing that is broken, so the recovery becomes muscle memory rather than something you half-remember from a blog. Pair it with the CKA exam guide for the overall plan and the free CKA practice questions for a quick self-check.
Continue with the CKA certificate management guide for the control-plane half of the PKI, the cluster access guide for the human-user CSR flow, the kubeadm installation guide for the join procedure this article sits underneath, and the CKA troubleshooting guide for the rest of the node-failure playbook.