Back to Blog

Kubelet TLS Bootstrapping & Certificate Rotation Internals for the CKA Exam: bootstrap-kubelet.conf, the Five Built-in CSR Signers, Auto-Approval RBAC (nodeclient vs selfnodeclient), kubelet-client-current.pem, Serving Certificates & serverTLSBootstrap, --cluster-signing-duration, the CSR Cleaner, Node Identity & NodeRestriction, and Repairing a Node Whose Certificate Expired

A practitioner's CKA guide to how a kubelet actually obtains, uses and renews its certificates — the bootstrap-kubelet.conf to kubelet.conf hand-off, the bootstrap token to system:bootstrappers to system:node identity chain, the five built-in signerName values and what each permits, the three kube-controller-manager controllers (approving, signing, cleaner) and the SubjectAccessReview-based auto-approval behind nodeclient and selfnodeclient, kubeadm's three ClusterRoleBindings, the kubelet-client-current.pem symlink, when rotation fires, why kubelet-serving CSRs are never auto-approved and what serverTLSBootstrap changes, --cluster-signing-duration and spec.expirationSeconds, CSR garbage collection timings, the Node authorizer and NodeRestriction label rules, ClusterTrustBundle and PodCertificateRequest (GA 1.37) awareness, and the exact recovery procedure for a node stuck on an expired kubelet client certificate.

By Sailor Team , November 11, 2026

Every kubeadm cluster you touch on the CKA exam has already done something remarkable before you log in: each worker joined with nothing but a 24-hour token, and a few seconds later it held a one-year X.509 client certificate issued by the cluster’s own CA, bound to an identity of the form system:node:<name>, with a controller standing by to renew it before it expires. Nobody ran openssl. Nobody approved anything. That is kubelet TLS bootstrapping, and it is the mechanism underneath kubeadm join, underneath a worker going NotReady because a certificate expired, underneath every Pending CSR you will ever see in kubectl get csr, and underneath the question of why kubectl logs works on one cluster and fails with an “unknown authority” error on another.

This guide is about that machinery. The existing CKA certificate management guide owns kubeadm certs check-expiration, kubeadm certs renew and control-plane certificate expiry, and it gives the kubelet certificate eight lines. The cluster access guide owns the human-user CSR workflow. Neither explains what the kubelet does with the CSR API on its own, which signer it addresses, why its renewal is approved automatically while a user’s never is, what the kubelet-client-current.pem symlink actually is, why the kubelet’s serving certificate is a different story from its client certificate, or how to repair a node whose rotation silently failed six months ago. That is this article. Everything below is checked against the Kubernetes v1.37 documentation, which is the version the live CKA runs on.

Where this sits on the CKA

The CKA weights are Cluster Architecture, Installation & Configuration 25%, Troubleshooting 30%, Services & Networking 20%, Workloads & Scheduling 15% and Storage 10%. Kubelet certificates sit squarely in the first two:

  • Cluster Architecture asks you to “create and manage Kubernetes clusters using kubeadm” and “manage the lifecycle of Kubernetes clusters”. Joining a node is TLS bootstrapping; the join token’s purpose is to authenticate a CSR.
  • Troubleshooting asks you to “troubleshoot clusters and nodes” and “troubleshoot cluster components”. A node that is NotReady with x509: certificate has expired or is not yet valid in the API server log is one of the canonical node-level failures, and the fix is not kubeadm certs renew.

You will not be asked to write a custom CSR approver on the exam. You will be expected to read kubectl get csr output and know what each column means, to recognise which CSRs approve themselves and which never will, and to recover a node whose kubelet cannot authenticate.

Two certificates, two directions

The kubelet holds two certificates with opposite jobs, and conflating them is the root of most confusion in this area:

CertificateDirectionWho verifies itDefault on kubeadmHow it is obtainedRotates?
Client certificatekubelet → kube-apiserver (every API call the kubelet makes)kube-apiserver, against --client-ca-fileIssued by the cluster CA via a CSR during joinTLS bootstrapping, signer kubernetes.io/kube-apiserver-client-kubeletYes, automatically (rotateCertificates: true)
Serving certificatekube-apiserver → kubelet (port 10250: logs, exec, port-forward, metrics)kube-apiserver only if --kubelet-certificate-authority is set; metrics-serverSelf-signed by the kubelet, written to --cert-dirSelf-signed, or a CSR with signer kubernetes.io/kubelet-serving when serverTLSBootstrap: trueOnly if serverTLSBootstrap is on, and then the CSR needs manual or custom approval

Hold those two rows in mind. The client certificate is what TLS bootstrapping is for. The serving certificate is the optional, less automated second act.

Why bootstrapping exists

The documentation lists the five jobs that fall on whoever runs a cluster with client-certificate authentication:

  1. Create the CA key and certificate.
  2. Distribute the CA certificate to the control plane nodes.
  3. Create a key and certificate for each kubelet, with a unique CN per node.
  4. Sign each kubelet certificate with the CA key.
  5. Distribute each key and signed certificate to the right node.

Steps 3 to 5 are the ones that scale badly, and they are the ones TLS bootstrapping automates. The idea, introduced with the certificate request and signing API in Kubernetes 1.4, is simple: give a new node a weak, short-lived credential that is good for exactly one thing, creating a CertificateSigningRequest, then let a controller approve and sign that request, and let the kubelet collect the result and write its own kubeconfig. After that the weak credential is never used again.

The bootstrap sequence, file by file

The documentation describes the sequence in fifteen steps. Here it is compressed, with the kubeadm file names you will actually find on an exam node.

  1. kubelet starts and looks for the file named by --kubeconfig. On kubeadm this is /etc/kubernetes/kubelet.conf. On a fresh node it does not exist yet.
  2. Falls back to --bootstrap-kubeconfig, which kubeadm writes as /etc/kubernetes/bootstrap-kubelet.conf during kubeadm join. This is an ordinary kubeconfig whose user credential is a bootstrap token, and whose cluster entry carries the CA certificate the join process discovered and verified.
  3. Authenticates with the token. The API server recognises the token (it is backed by a Secret of type bootstrap.kubernetes.io/token in kube-system, and --enable-bootstrap-token-auth=true is on) and authenticates the kubelet as system:bootstrap:<token-id> in the group system:bootstrappers plus any extra groups the token carries. On kubeadm the extra group is system:bootstrappers:kubeadm:default-node-token. The token format, Secret layout and expiry mechanics are covered in the CKS API server authentication guide; this article does not re-teach them.
  4. Generates a private key locally and creates a CSR object with spec.signerName: kubernetes.io/kube-apiserver-client-kubelet, subject CN=system:node:<nodeName>, O=system:nodes, and usages for client authentication.
  5. The CSR is approved, either automatically by the kube-controller-manager’s approving controller (the normal path) or by a human running kubectl certificate approve.
  6. The CSR is signed by the kube-controller-manager’s signing controller using the CA referenced by --cluster-signing-cert-file and --cluster-signing-key-file. The certificate lands in status.certificate.
  7. The kubelet collects the certificate, writes key and certificate into --cert-dir (default /var/lib/kubelet/pki), and writes a real kubeconfig to the --kubeconfig path. From now on kubelet.conf is used and bootstrap-kubelet.conf is ignored.
  8. Normal operation begins: the kubelet registers its Node object, starts reporting status, and is scheduled Pods.
  9. Optionally, rotation: when the certificate nears expiry the kubelet repeats steps 4 to 7 with its current certificate as the credential, and reconnects using the new one.

On an exam node you can see the artefacts of every step:

# The two kubeconfigs the kubelet knows about
sudo ls -l /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/kubelet.conf

# How the kubelet was told about them (kubeadm's systemd drop-in)
cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf | grep -E "bootstrap-kubeconfig|kubeconfig"

# What the bootstrapped kubeconfig points at
sudo grep -E "client-certificate|client-key" /etc/kubernetes/kubelet.conf
#   client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
#   client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

# The certificates themselves
sudo ls -l /var/lib/kubelet/pki/

That kubelet-client-current.pem line is the detail most candidates have never looked at, and it is the key to understanding rotation, so it gets its own section below.

The identity chain: token, bootstrapper, node

TLS bootstrapping is as much about identity as about certificates. Three identities appear in sequence, each more trusted than the last:

StageUsernameGroupsWhat it may do
Joiningsystem:bootstrap:<token-id>system:bootstrappers, system:bootstrappers:kubeadm:default-node-token, system:authenticatedCreate and read CSRs (ClusterRole system:node-bootstrapper); have kube-apiserver-client-kubelet CSRs auto-approved
Runningsystem:node:<nodeName>system:nodes, system:authenticatedEverything the Node authorizer allows for that node; renew its own client certificate; request a serving certificate
Human adminwhatever your kubeconfig saysfor kubeadm’s admin.conf: kubeadm:cluster-adminsApprove or deny any CSR, including the serving CSRs nobody else will

The second row is the one every other security control in the node path depends on. The Node authorizer only authorises a kubelet whose certificate puts it in group system:nodes with username exactly system:node:<nodeName>, and <nodeName> must match the registered Node name precisely. The NodeRestriction admission plugin uses the same identity to confine the kubelet to its own Node and its own Pods. The signer enforces the subject, so a kubelet cannot bootstrap itself into someone else’s identity. How the Node authorizer sits in the authorization chain is covered in the KCSA authorization chain guide.

The name-match requirement is also an exam trap: by default the kubelet uses the output of hostname as its node name, or --hostname-override, but with a cloud provider configured the provider may dictate the name. A kubelet whose certificate says system:node:worker-1 while it tries to update a Node called worker-1.example.internal is simply Forbidden.

The five built-in signers

Every CSR names a signer in spec.signerName, and the signer name decides which rules apply and whether anything will ever approve it automatically. Kubernetes ships five:

signerNamePurposeAuto-approved by kube-controller-manager?Permitted subjectSANsPermitted usages
kubernetes.io/kube-apiserver-clientClient certs for humans and toolsNeverAny (but CertificateSubjectRestriction admission rejects O=system:masters)HonouredMust include client auth; nothing beyond digital signature, key encipherment, client auth
kubernetes.io/kube-apiserver-client-kubeletKubelet client certsYes, when RBAC authorises the requesterO exactly ["system:nodes"], CN system:node:<nodeName>Forbidden (any SAN fails)digital signature + client auth, optionally key encipherment
kubernetes.io/kubelet-servingKubelet serving certsNeverO exactly ["system:nodes"], CN system:node:<nodeName>DNS and IP SANs honoured, at least one required; email and URI SANs forbiddendigital signature + server auth, optionally key encipherment
kubernetes.io/legacy-unknownNo trust guarantees; third-party distributionsNeverAnyHonouredAny
kubernetes.io/kube-apiserver-servingVerifying the API server’s own serving cert; trust bundle published as a ClusterTrustBundleNot handled by kube-controller-manager at allFollows DNS/IP SAN rulesAt least one DNS or IP SAN, must point at the API serverserver auth; recommended max lifetime 30 days

Three consequences worth memorising:

  • A user CSR (kube-apiserver-client) will sit Pending forever until an admin approves it. That is by design and it is why the cluster access guide’s workflow includes an explicit kubectl certificate approve.
  • A kubelet client CSR (kube-apiserver-client-kubelet) approves itself on a healthy kubeadm cluster. If you see one Pending for more than a few seconds, something is broken in the control plane, not in the kubelet.
  • A kubelet serving CSR (kubelet-serving) is supposed to sit Pending. The built-in approver refuses it on purpose, for the security reason explained below.

The legacy-unknown signer cannot even be requested through the stable certificates.k8s.io/v1 API; it exists for compatibility. The kube-apiserver-serving signer is new territory tied to ClusterTrustBundles, which went GA in 1.37, and is awareness-level for the CKA.

The three controllers inside kube-controller-manager

Nothing about bootstrapping lives in the API server beyond authenticating the request and storing the CSR object. The work happens in three controllers that kube-controller-manager --controllers=* enables by default. Their names in the --controllers list are certificatesigningrequest-approving-controller, certificatesigningrequest-signing-controller and certificatesigningrequest-cleaner-controller.

The approving controller

The approver only handles CSRs whose signer is kubernetes.io/kube-apiserver-client-kubelet. For each one it asks the API server a question via a SubjectAccessReview: is the user who created this CSR authorised to perform the virtual action this CSR represents? Two virtual resources exist, and they map to two ClusterRoles that ship with every cluster:

SituationVirtual subresource checkedClusterRole that grants itWho should hold it
A brand-new node requesting its first certificate, authenticated by a bootstrap tokencertificatesigningrequests/nodeclientsystem:certificates.k8s.io:certificatesigningrequests:nodeclientThe bootstrappers group
An existing node renewing its certificate, authenticated by its current certificatecertificatesigningrequests/selfnodeclientsystem:certificates.k8s.io:certificatesigningrequests:selfnodeclientsystem:nodes

The subtle part: selfnodeclient also verifies that the CSR’s subject matches the requester’s own identity. A node in system:nodes may renew the certificate for system:node:itself, not for another node. That is what the “self” means.

Two behaviours of the approver are exam-relevant. First, it never denies. An unauthorised request is simply ignored and stays Pending, which is why a missing RBAC binding looks like “nothing happens” rather than an error. Second, it is signer-specific: it does not touch kube-apiserver-client or kubelet-serving CSRs at all, regardless of RBAC.

The signing controller

The signer waits for the Approved condition and then issues the certificate with a local CA. It needs the CA material from --cluster-signing-cert-file and --cluster-signing-key-file (on kubeadm these point at /etc/kubernetes/pki/ca.crt and ca.key; in external-CA mode kubeadm sets both to empty strings and the controller signs nothing). You can alternatively give it a different CA per signer with --cluster-signing-kubelet-client-cert-file, --cluster-signing-kubelet-serving-cert-file, --cluster-signing-kube-apiserver-client-cert-file and --cluster-signing-legacy-unknown-cert-file (plus the matching -key-file flags). The generic and per-signer flags are mutually exclusive.

Lifetime is decided by --cluster-signing-duration, default 8760h0m0s (one year). Since Kubernetes 1.22 a CSR may also set spec.expirationSeconds, minimum 600 (ten minutes), and the issued lifetime is the minimum of the two. The flag is therefore a cap, not a fixed value: a requester can ask for shorter, never longer. Signing failures are reported only in kube-controller-manager logs, so a CSR that is Approved but never gets a certificate means you should read that Pod’s logs, not the kubelet’s.

The cleaner controller

CSR objects are not meant to accumulate. The garbage collector removes requests that have not changed state for a fixed period:

CSR stateDeleted after
Approved (and issued)1 hour
Denied1 hour
Failed1 hour
Pending24 hours
Any stateas soon as the issued certificate has expired

Exam implication: on a cluster that has been up for a day, kubectl get csr on a healthy system usually shows nothing, and that is normal. If a task says “a node joined an hour ago; inspect its CSR”, the object may already be gone. Conversely a kubelet-serving CSR you forgot to approve will vanish after 24 hours, and the kubelet will create a fresh one.

The RBAC that makes it automatic, and the bindings kubeadm creates

TLS bootstrapping is “just RBAC” once you know the three ClusterRoles involved. The documentation’s generic bindings and kubeadm’s actual bindings differ only in the subject group:

PurposeClusterRoleGeneric subject (docs)kubeadm ClusterRoleBinding namekubeadm subject
Let bootstrapping nodes create and read CSRssystem:node-bootstrappergroup system:bootstrapperskubeadm:kubelet-bootstrapgroup system:bootstrappers:kubeadm:default-node-token
Auto-approve first-time node client CSRssystem:certificates.k8s.io:certificatesigningrequests:nodeclientgroup system:bootstrapperskubeadm:node-autoapprove-bootstrapgroup system:bootstrappers:kubeadm:default-node-token
Auto-approve node client renewal CSRssystem:certificates.k8s.io:certificatesigningrequests:selfnodeclientgroup system:nodeskubeadm:node-autoapprove-certificate-rotationgroup system:nodes

Note that kubeadm binds the more specific bootstrappers group, system:bootstrappers:kubeadm:default-node-token, which is the auth-extra-groups value on the tokens kubeadm token create makes. A token created with a different extra group, or a static token file putting a user only in system:bootstrappers, would authenticate fine but would not be auto-approved on a stock kubeadm cluster. You can verify the bindings exist with one command:

kubectl get clusterrolebinding kubeadm:kubelet-bootstrap kubeadm:node-autoapprove-bootstrap kubeadm:node-autoapprove-certificate-rotation -o wide

For completeness, here is the renewal binding as the documentation writes it generically. On kubeadm you would never create it by hand, but recognising its shape on the exam is useful:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io

The CSR API also has its own fine-grained RBAC that explains three validating admission plugins you may meet in flag lists:

  • To create a CSR you need create, get, list, watch on certificatesigningrequests.
  • To approve one you need update on the certificatesigningrequests/approval subresource and the custom verb approve on resource signers with resourceName equal to the signer (kubernetes.io/kube-apiserver-client-kubelet) or a wildcard (kubernetes.io/*). The CertificateApproval admission plugin enforces the signer check.
  • To sign one you need update on certificatesigningrequests/status and the verb sign on signers. The CertificateSigning admission plugin enforces it.
  • CertificateSubjectRestriction rejects any kube-apiserver-client CSR whose organisation is system:masters, because that group is hard-wired to cluster-admin and bypasses RBAC.

What kubelet-client-current.pem really is

After bootstrapping, /var/lib/kubelet/pki/ on a kubeadm node looks like this:

$ sudo ls -l /var/lib/kubelet/pki/
kubelet-client-2026-11-11-09-14-02.pem
kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-11-11-09-14-02.pem
kubelet.crt
kubelet.key

Three facts about these files unlock the rest of the topic:

  • kubelet-client-<timestamp>.pem holds both the private key and the certificate in one PEM file. That is why kubelet.conf points client-certificate and client-key at the same path.
  • kubelet-client-current.pem is a symlink, not a file. Rotation works by writing a new timestamped file and atomically re-pointing the symlink. Because kubelet.conf references the symlink rather than embedding client-certificate-data, the kubeconfig never needs to change when the certificate does. (Clusters created with kubeadm older than 1.17 embedded the data and had to be edited by hand; modern kubeadm writes the symlink reference from the start.)
  • kubelet.crt and kubelet.key are the self-signed serving pair, generated because no tlsCertFile was provided. They have nothing to do with the client identity.

You can read the identity the API server sees straight out of the file:

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -issuer -dates -ext extendedKeyUsage
# subject=O = system:nodes, CN = system:node:worker-1
# issuer=CN = kubernetes
# notBefore=... notAfter=...   (one year apart by default)
# X509v3 Extended Key Usage: TLS Web Client Authentication

If the notAfter date is in the past, you have found your NotReady node’s problem.

Client certificate rotation: when it fires and what it needs

Rotation is governed by one KubeletConfiguration field:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
rotateCertificates: true

The field’s upstream default is false; kubeadm sets it to true in the kubelet-config ConfigMap it generates, which is why kubeadm nodes rotate without anyone configuring it. The equivalent flag --rotate-certificates is deprecated in favour of the config file, like most kubelet flags (the full precedence story is in the kubelet configuration guide).

When enabled, the kubelet’s certificate manager:

  1. Watches the current certificate’s lifetime and picks a renewal moment when between 30% and 10% of the lifetime remains, that is, somewhere between 70% and 90% elapsed, randomised so that a fleet of nodes does not all renew in the same second. For a one-year certificate that is roughly month nine to month eleven.
  2. Generates a new private key (keys rotate with certificates; it does not re-sign the old key).
  3. Creates a CSR with signer kubernetes.io/kube-apiserver-client-kubelet, authenticated with its current certificate, so the requester is system:node:<name> in group system:nodes.
  4. Relies on the selfnodeclient binding to be auto-approved, then on the signing controller to issue.
  5. Writes kubelet-client-<newtimestamp>.pem, moves the symlink, and reconnects its API server connections using the new certificate.

Because step 3 uses the current certificate, rotation only works while that certificate is still valid. A kubelet that was powered off for a year, or whose control plane was down at every renewal attempt, comes back holding an expired certificate and cannot authenticate to request a new one. That is the failure mode the recovery section fixes, and it is the only situation where you do this by hand.

Two kubelet metrics tell you whether rotation is healthy without reading logs: kubelet_certificate_manager_client_ttl_seconds (seconds until the client certificate expires; negative means already expired; +Inf means unused or invalid) and kubelet_certificate_manager_client_expiration_renew_errors (a counter that should stay at zero). The CIS Kubernetes Benchmark checks that --rotate-certificates is not set to false, so turning rotation off is both operationally and compliance-wise wrong.

The serving certificate and serverTLSBootstrap

Now the second certificate. By default the kubelet self-signs a serving certificate for its HTTPS endpoint on port 10250, saved as kubelet.crt and kubelet.key in --cert-dir, because no tlsCertFile and tlsPrivateKeyFile were supplied. Three parties connect to that endpoint: the API server (for kubectl logs, exec, attach, port-forward, cp), metrics-server (for kubectl top), and anything else scraping kubelet metrics.

Here is the part that surprises people: by default the API server does not verify the kubelet’s serving certificate at all. It authenticates itself to the kubelet with --kubelet-client-certificate and --kubelet-client-key, but it only verifies the kubelet’s identity if --kubelet-certificate-authority is set, which kubeadm does not do, precisely because the kubelets’ certificates are self-signed and could not be verified. The CIS Benchmark flags this (the API server check for --kubelet-certificate-authority), and the fix requires giving kubelets real serving certificates first.

That is what serverTLSBootstrap is for:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
rotateCertificates: true

With serverTLSBootstrap: true, instead of self-signing, the kubelet creates a CSR with signer kubernetes.io/kubelet-serving, subject O=system:nodes, CN=system:node:<name>, with its hostname and IP addresses as DNS and IP SANs, and usages for server authentication. The field requires the RotateKubeletServerCertificate feature gate, which has been beta and enabled by default since 1.12, so on any cluster you meet it is on. Enabling the gate makes the kubelet both request the serving certificate after client bootstrapping and rotate it; the deprecated flag equivalent is --rotate-server-certificates.

And then the CSR sits there:

$ kubectl get csr
NAME        AGE     SIGNERNAME                        REQUESTOR                      REQUESTEDDURATION   CONDITION
csr-9wvgt   112s    kubernetes.io/kubelet-serving     system:node:worker-1           <none>              Pending
csr-lz97v   1m58s   kubernetes.io/kubelet-serving     system:node:control-plane-1    <none>              Pending

The core approving controller deliberately refuses to approve kubelet-serving CSRs, for a documented security reason: a kubelet (or anyone holding a kubelet’s client certificate) could request a serving certificate with any DNS name or IP in the SANs, including the API server’s, and nothing in the request itself proves the node owns those addresses. Automatic approval would turn a single compromised node certificate into a certificate-minting facility. So approval must come from a human (kubectl certificate approve csr-9wvgt) or from a custom approver controller that verifies, out of band, that the requested SANs really belong to that node. The documentation’s checklist for any such approver is worth knowing because exam distractors are built from it: approve only CSRs whose spec.username is system:node:<nodeName> and whose spec.groups contains system:nodes, whose usages contain server auth (optionally digital signature and key encipherment) and nothing else, and whose SANs are only IP and DNS names belonging to the requesting node, with no URI or email SANs.

Once approved, the kubelet writes kubelet-server-current.pem (another symlink), and the serving certificate expires after one year like everything else. Because rotateCertificates is true, a new serving CSR appears before expiry and must be approved again, every year, for every node. Forgetting that is how clusters that “worked for a year” suddenly lose kubectl logs. The metric kubelet_certificate_manager_server_ttl_seconds reports the shortest serving-certificate TTL and kubelet_server_expiration_renew_errors counts failures.

To enable this on an existing kubeadm cluster, the documented procedure is: edit the kubelet-config ConfigMap in kube-system to add serverTLSBootstrap: true (so future joins and kubeadm upgrade node pick it up), add the same field to /var/lib/kubelet/config.yaml on each node, systemctl restart kubelet, approve the resulting CSRs, and only then add --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt to the API server manifest. Do it in the other order and kubectl logs breaks cluster-wide with “x509: certificate signed by unknown authority” until every node has a CA-signed serving certificate.

—cluster-signing-duration and short-lived certificates

Everything the built-in signers issue defaults to one year because --cluster-signing-duration defaults to 8760h0m0s. Operators who want shorter kubelet certificates change that flag in /etc/kubernetes/manifests/kube-controller-manager.yaml; the kubelet restarts the static Pod and every certificate issued from then on is shorter. Existing certificates keep their original expiry until they rotate.

Shorter means more CSR churn (each node rotates at 70 to 90% of a shorter window), which is harmless for client certificates because they auto-approve, but becomes a manual approval treadmill for serving certificates unless you run an approver. A requester can also ask for less than the cap with spec.expirationSeconds; the kubelet-serving and kube-apiserver-client-kubelet CSRs the kubelet generates do not set it (that is the <none> in the REQUESTEDDURATION column), so they always receive the full cap.

The kubeadm certs commands never touch any of this. kubeadm certs check-expiration reports control-plane certificates and the .conf kubeconfigs and explicitly leaves kubelet.conf out, “because kubeadm configures kubelet for automatic certificate renewal with rotatable certificates under /var/lib/kubelet/pki”. If an exam task asks you to inspect kubelet certificate expiry, the tool is openssl on the PEM file, not kubeadm.

What about kube-proxy and other node components?

TLS bootstrapping is a kubelet feature. kube-proxy also talks to the API server from every node, yet it never appears in kubectl get csr. The documentation offers two routes for such components: distribute certificates the old way, or run them as a DaemonSet with a ServiceAccount, which is what kubeadm does. kube-proxy runs in kube-system as a DaemonSet and authenticates with a projected ServiceAccount token, so its identity comes from the token machinery (owned by the CKS ServiceAccount token guide), not from certificates. CNI agents, node exporters and CSI node plugins follow the same pattern. The only process on a node that must use the CSR flow is the kubelet, because it exists before any Pod can.

Node identity enforcement: the Node authorizer and NodeRestriction

The bootstrapping signer’s strict subject rules only matter because two control-plane components rely on that subject.

The Node authorizer (--authorization-mode=Node,RBAC, which kubeadm sets, or type: Node in a structured AuthorizationConfiguration) grants a kubelet exactly what it needs: read services, endpoints, nodes and pods; read the Secrets, ConfigMaps, PVCs and PVs referenced by Pods bound to its node; write its own node and node status, pods and pod status, and events; use the CSR API for bootstrapping; and create TokenReviews and SubjectAccessReviews. With the AuthorizeNodeWithSelectors behaviour, kubelets are limited to reading their own Node object and only the Pods bound to them. None of this applies unless the identity is system:node:<name> in system:nodes.

The NodeRestriction admission plugin (--enable-admission-plugins=NodeRestriction, on by default in kubeadm) then confines writes:

  • A kubelet may modify only its own Node object and only Pods bound to it, may not delete its Node, and may not add or remove taints.
  • Labels under kubernetes.io/ and k8s.io/ are policed. Forbidden for kubelets: anything prefixed node-restriction.kubernetes.io/ (reserved for administrators to pin workloads to trusted nodes) and node-role.kubernetes.io/ (so a node cannot declare itself a control-plane node). Allowed: kubernetes.io/hostname, kubernetes.io/arch, kubernetes.io/os, node.kubernetes.io/instance-type, topology.kubernetes.io/region and /zone, and anything under kubelet.kubernetes.io/ or node.kubernetes.io/. Everything else under those prefixes is reserved.
  • With ServiceAccountNodeAudienceRestriction, a kubelet may only request ServiceAccount tokens for audiences already referenced by Pods on its node, unless RBAC grants the custom verb request-serviceaccounts-token-audience.

The CKA angle: if a task asks you to label a node node-role.kubernetes.io/worker= you do it as an admin with kubectl label, which is fine. If you try to achieve the same by adding --node-labels=node-role.kubernetes.io/worker= to the kubelet’s flags, the kubelet’s own registration is rejected and the node never becomes Ready. Likewise, the error nodes "worker-2" is forbidden: node "worker-1" is not allowed to modify node "worker-2" in a kubelet log means the certificate identity and the node name disagree.

Beyond kubelets: ClusterTrustBundle and PodCertificateRequest (awareness)

Two newer certificates.k8s.io/v1 kinds reached GA in Kubernetes 1.37 and reuse the signer concept. Neither is CKA core, but both appear in kubectl api-resources output on a current cluster and recognising them avoids confusion:

  • ClusterTrustBundle is a cluster-scoped object that distributes X.509 trust anchors (CA certificates) to workloads. Signer-linked bundles are named <signerDomain>:<signerPath>:<name> and can only be written by a controller holding the custom verb attest on that signer; signer-unlinked bundles have no colon in their name and plain RBAC. All ServiceAccounts can read them by default, and Pods consume them through the clusterTrustBundle projected volume source. The kube-controller-manager’s kube-apiserver-serving-clustertrustbundle-publisher-controller publishes the bundle for the kubernetes.io/kube-apiserver-serving signer. (The feature gate was alpha 1.27, beta off by default 1.33, GA 1.37.)
  • PodCertificateRequest is a CSR-like object the kubelet creates on a Pod’s behalf when the Pod uses a podCertificate projected volume. It carries podName, podUID, serviceAccountName, nodeName, a maxExpirationSeconds (default 24 hours) and a stub PKCS#10 request holding just the public key. Unlike CSRs there is no approval phase: the signer’s controller issues, denies or fails the request directly, writes the chain into status.certificateChain plus notBefore, notAfter and beginRefreshAt, and the kubelet handles refresh. NodeRestriction ensures a node can only create requests for Pods actually running on it, and a kube-controller-manager cleaner deletes requests older than 15 minutes. (Alpha 1.34, beta 1.35, GA 1.37.)

Exam value: if you see podcertificaterequests in kubectl get csr’s neighbourhood, it is workload identity, not node identity, and it never needs kubectl certificate approve.

Repairing a node whose kubelet certificate expired

This is the one hands-on recovery procedure in the topic, taken from the kubeadm troubleshooting documentation, and it is a plausible Troubleshooting task because the symptoms are so specific:

  • kubectl get nodes shows the node NotReady.
  • journalctl -u kubelet on the node shows Unauthorized or x509: certificate has expired or is not yet valid when talking to the API server, and the kube-apiserver log shows the same x509 message for that client.
  • openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates confirms notAfter is in the past.

Rotation cannot self-heal here because the kubelet’s only credential is the expired certificate. The documented fix re-bootstraps the node with a fresh kubeconfig signed by the CA:

# 1. On the FAILED node: back up and remove the stale kubeconfig and client certs
sudo mkdir -p /root/kubelet-backup
sudo mv /etc/kubernetes/kubelet.conf /root/kubelet-backup/
sudo mv /var/lib/kubelet/pki/kubelet-client* /root/kubelet-backup/

# 2. On a WORKING control-plane node that has /etc/kubernetes/pki/ca.key:
#    mint a kubeconfig whose embedded cert is CN=system:node:<NODE>, O=system:nodes
NODE=worker-1
sudo kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE > kubelet.conf
#    If the server/cluster name inside is not right for your cluster, fix them,
#    or pass --config with the correct ClusterConfiguration.

# 3. Copy kubelet.conf to the failed node as /etc/kubernetes/kubelet.conf

# 4. On the FAILED node: restart and wait for the symlink to be recreated
sudo systemctl restart kubelet
sudo ls -l /var/lib/kubelet/pki/kubelet-client-current.pem

# 5. Point kubelet.conf back at the rotatable symlink: replace
#    client-certificate-data / client-key-data with
#      client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
#      client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
#    then
sudo systemctl restart kubelet

# 6. Verify
kubectl get nodes

Why step 5 matters: the kubeconfig kubeadm kubeconfig user emits embeds a certificate. If you leave it that way, the kubelet will happily bootstrap a rotatable certificate into /var/lib/kubelet/pki but keep authenticating with the embedded one, which will expire again in a year with no rotation. Re-pointing the kubeconfig at the symlink restores the self-healing loop. If your control plane has no ca.key (external CA mode), you must sign the embedded certificate externally instead.

What you should not do: kubeadm certs renew all. It renews control-plane certificates and the admin, controller-manager and scheduler kubeconfigs, and does nothing for kubelet.conf or /var/lib/kubelet/pki. Running it is harmless but wastes exam minutes.

Diagnosis table

SymptomWhere you see itLikely causeFix
kubeadm join fails with Unauthorizedjoin outputBootstrap token expired (default TTL 24 h) or deletedkubeadm token create --print-join-command on a control plane node
kubeadm join fails with certificate signed by unknown authorityjoin outputWrong --discovery-token-ca-cert-hash or wrong CA in bootstrap kubeconfigRecompute the hash from /etc/kubernetes/pki/ca.crt; re-run join
Node never appears; kubelet log csr ... is Pending for minuteskubelet log, kubectl get csrApprover not running, or no nodeclient binding for the token’s groupCheck kube-controller-manager Pod and logs; check kubeadm:node-autoapprove-bootstrap binding; as a stopgap kubectl certificate approve
CSR Approved but no certificate, kubelet keeps waitingkubectl get csr shows Approved without IssuedSigning controller cannot sign: missing or empty --cluster-signing-*-file, external CA modeRead kube-controller-manager logs; signing failures appear only there
kubectl get csr is empty although a node joined an hour agokubectl get csrCleaner deleted the approved CSR after 1 hourNormal; nothing to fix
kubelet-serving CSRs stay Pendingkubectl get csrBy design: never auto-approvedkubectl certificate approve <name> or run a SAN-verifying approver
kubectl logs and exec fail cluster-wide with x509: certificate signed by unknown authoritykubectl output--kubelet-certificate-authority set on the API server while kubelets still self-signApprove serving CSRs first, or remove the flag until every node has a signed serving cert
kubectl top nodes shows <unknown> or metrics-server logs TLS errorsmetrics-server logsmetrics-server cannot verify self-signed kubelet certificatesEnable serverTLSBootstrap and approve serving CSRs, or configure metrics-server to skip kubelet TLS verification (less secure)
Node NotReady; API server log x509: certificate has expired or is not yet valid for that clientkube-apiserver log, journalctl -u kubeletKubelet client cert expired and rotation could not runRe-bootstrap with kubeadm kubeconfig user --org system:nodes --client-name system:node:<NODE> as above
Node NotReady; kubelet log node "X" is not allowed to modify node "Y"kubelet logCertificate CN does not match the registered node name (--hostname-override, cloud provider naming)Make the node name match the certificate, or re-bootstrap with the right --client-name
Kubelet rejected when registering with --node-labels=node-role.kubernetes.io/...kubelet logNodeRestriction forbids kubelets from setting node-role.kubernetes.io/ and node-restriction.kubernetes.io/ labelsRemove the flag; label from an admin kubeconfig
A user’s CSR is never approvedkubectl get csrkube-apiserver-client signer is never auto-approvedExpected; approve manually (cluster access guide)
A CSR fails or the signer refuses it with a subject errorkubectl describe csr, kube-controller-manager logO not exactly system:nodes, CN not system:node:<name>, or a SAN on a kube-apiserver-client-kubelet requestRegenerate the request with the right subject and no SANs
After kubeadm certs renew all, node still NotReadykubectl get nodesKubelet certs are not managed by kubeadm certsFollow the kubelet recovery procedure instead

Hands-on: read the whole chain on a kubeadm cluster in ten minutes

Run this on any kubeadm practice cluster; nothing here is destructive except the optional serving-certificate step, which is reversible.

# 1. The kubelet's two kubeconfigs and how systemd passes them
cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
sudo grep -E "server:|client-certificate|client-key" /etc/kubernetes/kubelet.conf

# 2. The client certificate identity and expiry
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -dates -ext extendedKeyUsage

# 3. The rotation setting kubeadm wrote
grep -E "rotateCertificates|serverTLSBootstrap" /var/lib/kubelet/config.yaml
kubectl -n kube-system get cm kubelet-config -o yaml | grep -E "rotateCertificates|serverTLSBootstrap"

# 4. The RBAC that auto-approves bootstrap and rotation
kubectl get clusterrolebinding kubeadm:kubelet-bootstrap kubeadm:node-autoapprove-bootstrap kubeadm:node-autoapprove-certificate-rotation -o wide
kubectl describe clusterrole system:node-bootstrapper system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

# 5. The signing cap on the controller manager
sudo grep cluster-signing /etc/kubernetes/manifests/kube-controller-manager.yaml

# 6. Does the API server verify kubelet serving certs?
sudo grep -E "kubelet-certificate-authority|kubelet-client-certificate" /etc/kubernetes/manifests/kube-apiserver.yaml

# 7. (Optional, reversible) Request a signed serving cert on ONE node
sudo sed -i 's/^serverTLSBootstrap:.*//' /var/lib/kubelet/config.yaml
echo "serverTLSBootstrap: true" | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
kubectl get csr                       # a kubernetes.io/kubelet-serving CSR appears, Pending
kubectl certificate approve <csr-name>
sudo ls -l /var/lib/kubelet/pki/      # kubelet-server-current.pem symlink now exists

# 8. Watch a CSR get garbage-collected after an hour, or just note the timings
kubectl get csr -o custom-columns=NAME:.metadata.name,SIGNER:.spec.signerName,REQUESTOR:.spec.username,AGE:.metadata.creationTimestamp

If you want to practise the recovery procedure, do it on a disposable worker: move kubelet.conf and the kubelet-client* files aside, watch the node go NotReady, then follow the six steps above. Doing it once makes the exam version a five-minute task instead of a twenty-minute panic.

Scenario drill

ScenarioAnswer
A new worker runs kubeadm join. Which signer does its first CSR address?kubernetes.io/kube-apiserver-client-kubelet
Who approves that CSR on a stock kubeadm cluster?kube-controller-manager’s approving controller, via a SubjectAccessReview against the nodeclient virtual subresource
Which group must the joining identity be in for kubeadm’s auto-approve binding to match?system:bootstrappers:kubeadm:default-node-token
A year later the same kubelet renews. Which ClusterRole authorises the auto-approval?system:certificates.k8s.io:certificatesigningrequests:selfnodeclient, bound to system:nodes
When does the kubelet begin renewal?At a random point when 30% to 10% of the certificate lifetime remains
kubelet.conf references client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem. What is that file?A symlink to the latest timestamped PEM containing key and certificate; rotation re-points it
Default issued lifetime, and the flag that caps it?One year; --cluster-signing-duration default 8760h0m0s
A CSR sets spec.expirationSeconds: 300. Result?Rejected: minimum is 600 seconds
A CSR sets spec.expirationSeconds: 86400 with the default cap. Issued lifetime?24 hours, the minimum of the request and the cap
kubectl get csr shows kubernetes.io/kubelet-serving CSRs Pending for 20 minutes. Bug?No; the built-in approver never approves serving CSRs; approve manually or run a verifying approver
Why does the built-in approver refuse serving CSRs?The request could carry any DNS or IP SAN, so automatic approval would let a compromised node mint certificates for arbitrary names
Which KubeletConfiguration field makes the kubelet request a signed serving certificate?serverTLSBootstrap: true (requires RotateKubeletServerCertificate, beta on by default since 1.12)
By default, does kube-apiserver verify the kubelet’s serving certificate?No; only when --kubelet-certificate-authority is set, which kubeadm does not do
How long does an approved CSR stay in the cluster?1 hour (denied and failed also 1 hour; pending 24 hours; any CSR is removed once its certificate expires)
A kubelet client CSR includes a DNS SAN. Outcome?Refused by the kube-apiserver-client-kubelet signer, which forbids SANs
A user CSR requests O=system:masters. Outcome?Rejected by the CertificateSubjectRestriction admission plugin
Node NotReady, API server logs x509: certificate has expired, kubelet cert notAfter in the past. Fix?Re-bootstrap: remove kubelet.conf and kubelet-client*, generate a kubeconfig with kubeadm kubeconfig user --org system:nodes --client-name system:node:<NODE>, restart, then point the kubeconfig at the symlink
Does kubeadm certs renew all fix that node?No; kubeadm does not manage kubelet client certificates
A kubelet starts with --node-labels=node-role.kubernetes.io/infra=. Result?NodeRestriction rejects the registration; node-role.kubernetes.io/ is forbidden for kubelets
Kubelet log: node "w1" is not allowed to modify node "w1.internal". Cause?Certificate CN system:node:w1 does not match the registered node name
kube-proxy never shows up in kubectl get csr. Why?It runs as a DaemonSet with a ServiceAccount token, not with a bootstrapped certificate
Which object distributes CA trust anchors to Pods in 1.37?ClusterTrustBundle (GA 1.37), consumed via a clusterTrustBundle projected volume
Which object does the kubelet create for a Pod using a podCertificate projected volume, and does it need approval?PodCertificateRequest (GA 1.37); no approval phase, the signer issues or denies directly, and requests are cleaned up after 15 minutes

Mistakes that cost points

  • Treating kubeadm certs renew as the fix for a kubelet certificate problem. It renews control-plane certificates and three kubeconfigs. kubelet.conf is deliberately excluded because the kubelet rotates its own.
  • Approving every Pending CSR reflexively. On the exam a kube-apiserver-client CSR for a user should be approved when the task says so; a kubelet-serving CSR should be approved after checking the requestor is system:node:<name>; a kube-apiserver-client-kubelet CSR stuck in Pending means the control plane is broken and approving it by hand hides the real fault.
  • Forgetting that rotation needs a valid current certificate. Once it has expired, no amount of restarting the kubelet will renew it.
  • Enabling --kubelet-certificate-authority before kubelets have signed serving certificates, which breaks logs and exec cluster-wide.
  • Assuming serving certificates rotate silently like client certificates. They rotate, but each rotation produces a CSR that again needs approval.
  • Looking for kubelet.conf expiry in kubeadm certs check-expiration. It is not listed; use openssl on kubelet-client-current.pem.
  • Setting node-role labels from the kubelet. NodeRestriction forbids it; label from an admin context.
  • Reading kubelet logs for a signing problem. Signing failures are reported only in kube-controller-manager logs.

Frequently Asked Questions

Does the kubelet’s client certificate really renew itself with no human involvement?

Yes, on a kubeadm cluster. kubeadm sets rotateCertificates: true, and the renewal CSR is approved automatically because the requester is in group system:nodes and kubeadm’s kubeadm:node-autoapprove-certificate-rotation binding grants the selfnodeclient ClusterRole. The signing controller issues the certificate, the kubelet swaps the kubelet-client-current.pem symlink, and it reconnects. The only prerequisite is that the current certificate is still valid when the kubelet tries, which is why a long-powered-off node needs the manual recovery procedure.

Why does a user’s CSR need kubectl certificate approve but a node’s does not?

Because they address different signers. kubernetes.io/kube-apiserver-client is never auto-approved by design, since the subject can be anything and a wrongly issued certificate could be cluster-admin. kubernetes.io/kube-apiserver-client-kubelet is auto-approved only when a SubjectAccessReview confirms the requester is authorised for nodeclient or selfnodeclient, and the signer enforces that the subject is exactly O=system:nodes, CN=system:node:<name> with no SANs, so the worst a successful request can obtain is a node identity for the requester itself.

Why are kubelet serving CSRs left Pending on every cluster I have seen?

The core approving controller refuses kubernetes.io/kubelet-serving requests for a documented security reason: the CSR can request any DNS or IP SAN, and nothing in the cluster can automatically prove the node owns those addresses. Approval has to come from an administrator or from a custom controller that verifies the SANs out of band. Most kubeadm clusters leave serverTLSBootstrap off and live with self-signed serving certificates, which is why the API server does not verify them by default.

What is the difference between rotateCertificates and serverTLSBootstrap?

rotateCertificates controls renewal of the client certificate the kubelet presents to the API server and is true on kubeadm. serverTLSBootstrap tells the kubelet to obtain its serving certificate for port 10250 from the CSR API instead of self-signing, and is false by default. With both on, the serving certificate also rotates, but every rotation creates a CSR that needs approval.

How long are the issued certificates valid, and can I change it?

One year by default, from --cluster-signing-duration=8760h0m0s on kube-controller-manager. Lower it in /etc/kubernetes/manifests/kube-controller-manager.yaml to shorten every certificate issued after the restart. Requesters may also ask for less with spec.expirationSeconds (minimum 600 seconds); the issued lifetime is the smaller of the two values.

My node is NotReady and the API server says the kubelet’s certificate has expired. Is there a quick fix?

Yes, and it is not kubeadm certs renew. Back up and remove /etc/kubernetes/kubelet.conf and /var/lib/kubelet/pki/kubelet-client*, generate a new kubeconfig on a control plane node with kubeadm kubeconfig user --org system:nodes --client-name system:node:<NODE>, copy it to the node, restart the kubelet, then edit the kubeconfig to reference /var/lib/kubelet/pki/kubelet-client-current.pem for both certificate and key and restart again. The last step restores automatic rotation; skipping it leaves you with an embedded certificate that will expire again.

Are ClusterTrustBundle and PodCertificateRequest on the CKA?

They are not named in the curriculum, but both are GA in Kubernetes 1.37, the version the exam runs on, and both live in the same certificates.k8s.io API group as CSRs. Know what they are: ClusterTrustBundle distributes CA certificates to workloads, PodCertificateRequest provisions workload certificates through a kubelet-managed projected volume with no approval phase. Neither replaces kubelet TLS bootstrapping for node identity.

Conclusion

Kubelet TLS bootstrapping is the mechanism that turns a 24-hour join token into a one-year node identity and keeps that identity alive without an administrator: a bootstrap kubeconfig that is only good for creating a CSR, a signer that only issues system:node:<name> client certificates with no SANs, an approver that says yes only after a SubjectAccessReview against nodeclient or selfnodeclient, a one-year cap from --cluster-signing-duration, a symlink that lets the kubeconfig stay constant while the certificate changes, and a cleaner that tidies the evidence away after an hour. The serving certificate is the deliberate exception: it is self-signed unless you ask for serverTLSBootstrap, and even then every CSR waits for a human or a verifying controller, because automatic approval would be a certificate-minting hole. Once you can read kubectl get csr and /var/lib/kubelet/pki fluently, the scary exam symptom, a NotReady node with an expired certificate, becomes a six-step procedure you have already rehearsed.

If you want to rehearse it under exam conditions, Sailor.sh’s CKA Certification-Ready Mock Exam Bundle gives you real kubeadm clusters in the browser across five timed, performance-based mock exams with explanations, including node-level troubleshooting scenarios where the certificate chain is the thing that is broken, so the recovery becomes muscle memory rather than something you half-remember from a blog. Pair it with the CKA exam guide for the overall plan and the free CKA practice questions for a quick self-check.

Continue with the CKA certificate management guide for the control-plane half of the PKI, the cluster access guide for the human-user CSR flow, the kubeadm installation guide for the join procedure this article sits underneath, and the CKA troubleshooting guide for the rest of the node-failure playbook.

Limited Time Offer: Get 80% off all Mock Exam Bundles | Sale ends in 7 days. Start learning today.

Claim Now