Every CKAD task starts the same way: a container has to come from somewhere. Most candidates think about that step for exactly as long as it takes to type image: nginx, and then lose ten minutes when the Pod sits in ImagePullBackOff because the exam cluster has the image cached locally under a tag the kubelet refuses to trust, or because the private registry Secret is in the wrong namespace, or because args: ["$(PORT)"] printed the literal string $(PORT). None of those are Kubernetes being difficult. They are the kubelet following rules that the documentation states precisely and almost nobody reads.
This guide reads them for you. Our container images guide covers Dockerfiles, tags and building; the ConfigMaps & Secrets guide covers creating Secrets including the Docker registry type; and the application troubleshooting guide covers the general diagnosis loop. This post sits between them and goes inside the pull: how the kubelet decides whether to pull, how it authenticates, what it does when the pull fails, how a newer Kubernetes mounts an image as a volume, and how command and args are resolved once the image is finally on the node.
Where This Lands in the CKAD Curriculum
Image pulling touches three of the five CKAD domains. Application Design and Build (20%) owns “define, build and modify container images” and the Pod spec itself. Application Environment, Configuration and Security (25%) owns Secrets, ServiceAccounts and the private-registry pattern. Application Observability and Maintenance (15%) owns debugging, and ImagePullBackOff is the single most common Waiting reason you will diagnose. The exam environment is a kubeadm cluster running a recent Kubernetes minor release (check the CNCF curriculum page for the exact version at your exam date), so everything below is written against the current documentation, with the version in which each feature changed noted where it matters.
Image References: What the Kubelet Actually Parses
A container image reference has four parts:
registry.example.com:5000/team/app:1.4.2@sha256:3f1a...
└────── registry ──────┘└─ path ─┘└tag┘└───── digest ─────┘
The rules the kubelet (via the container runtime) applies:
| Part | If omitted | Notes |
|---|---|---|
| Registry | The runtime’s default public registry, docker.io | Can include a port. The default registry is a container runtime setting, not a Kubernetes one |
| Path | — | busybox expands to docker.io/library/busybox, so official Docker Hub images live under library/ |
| Tag | latest | Must match [a-zA-Z0-9_][a-zA-Z0-9._-]{0,127} (up to 128 characters) |
| Digest | — | sha256:<hex>; immutable. If both a tag and a digest are present, only the digest is used |
A tag is a moveable pointer: a registry can re-point app:1.4.2 to a different image after you deployed it, which is exactly why a Deployment rolled out across 50 nodes over an hour can end up running two different builds under one tag. A digest is content-addressed and cannot move. The documentation’s advice is blunt: do not use :latest in production, prefer a meaningful tag and ideally a digest, and consider an admission controller that rewrites tags to digests. For the exam, you will mostly see tags; know what the absence of one does to the pull policy.
imagePullPolicy: The Four Default Rules
imagePullPolicy has three values:
| Value | What the kubelet does |
|---|---|
IfNotPresent | Pull only if the image is not already present on the node |
Always | Ask the runtime to pull every time the container starts. The runtime contacts the registry, resolves the tag to a digest, and downloads only the layers it does not already have |
Never | Never pull. If the image is not on the node, the container fails to start (ErrImageNeverPull) |
The part that fails candidates is what happens when you omit the field. The default is derived from the image reference at the moment the Pod object is created:
| Image reference when the Pod is created | Default imagePullPolicy |
|---|---|
| Has a digest | IfNotPresent |
Tag is :latest | Always |
No tag at all (implies latest) | Always |
Any other tag (1.4.2, stable, v2-rc1) | IfNotPresent |
The same rule applies to Pod templates in Deployments, StatefulSets, Jobs and everything else. Three traps follow directly.
Trap 1: latest means Always, and Always means the registry must be reachable
In the exam environment, images are typically pre-loaded on nodes. image: myapp:latest (or image: myapp) defaults to Always, so the kubelet asks the runtime to check the registry. If myapp exists only locally and there is no registry to consult, the pull fails and you get ImagePullBackOff even though the image is right there. The container images guide flagged this; the mechanism is the default table above. Fix: use an explicit non-latest tag, or set imagePullPolicy: IfNotPresent (or Never) explicitly.
Trap 2: The policy freezes at creation
The default is computed once and stored in the object. Change a Deployment’s image from app:1.4.2 (default IfNotPresent) to app:latest with kubectl set image, and the stored policy stays IfNotPresent. Kubernetes does not recompute it. Nodes that already have an old app:latest cached will happily run the stale image forever. If you want Always, set it explicitly. You can read what was stored with:
kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[*].imagePullPolicy}{"\n"}'
Trap 3: Always is cheaper than it sounds, but not free
Because the runtime caches layers, Always on an image that has not changed costs one registry round trip (manifest fetch and digest comparison), not a full download. The documentation calls Always “efficient” as long as the registry is reliably reachable. The reverse is also true: a registry outage with Always means new Pods cannot start even on nodes that have the image, which is a resilience consideration the exam occasionally probes.
Forcing pulls cluster-wide
Three ways to force a pull: set Always explicitly; omit the policy and use :latest/no tag; or have the cluster administrator enable the AlwaysPullImages admission controller, which rewrites every Pod’s policy to Always. That last one is the multi-tenant security answer: it guarantees a Pod cannot use a private image that happens to be cached on the node without presenting credentials for it. Keep it in mind for the credential-verification section below, which is the modern alternative.
Setting it imperatively
kubectl run web --image=nginx:1.27 --image-pull-policy=IfNotPresent --dry-run=client -o yaml > web.yaml
kubectl explain pod.spec.containers.imagePullPolicy
kubectl explain prints the exact default rule text when you forget it under pressure.
What Happens When the Pull Fails: ErrImagePull and ImagePullBackOff
A failed pull leaves the container in the Waiting state. The first failure shows reason ErrImagePull; the kubelet then backs off and the reason becomes ImagePullBackOff while it waits to retry. The retry delay grows exponentially and is capped at a compiled-in 300 seconds (5 minutes). The kubelet keeps retrying forever; there is no “give up”.
That has a practical consequence: fix the cause and the Pod can still show ImagePullBackOff for up to five minutes while the current back-off period runs out. Deleting the Pod (or the ReplicaSet recreating it) resets the timer, which is why kubectl delete pod “fixes” things after you corrected the image name. On a Deployment, kubectl rollout restart does the same.
Read the real cause in the events, not the reason:
kubectl describe pod web | sed -n '/Events:/,$p'
kubectl get events --field-selector involvedObject.name=web --sort-by=.lastTimestamp
| Event message fragment | Actual cause | Fix |
|---|---|---|
manifest unknown / not found | Tag or path does not exist | Fix the tag (nginx:1.27 not nginx:1.27.0.1) |
pull access denied / unauthorized / no basic auth credentials | Private image, no or wrong credentials | Create a docker-registry Secret, add imagePullSecrets |
FailedToRetrieveImagePullSecret | Secret named in imagePullSecrets does not exist in the Pod’s namespace | Create it in the right namespace or fix the name |
x509: certificate signed by unknown authority | Registry uses a CA the node does not trust | Node-level runtime config (not a Pod fix) |
dial tcp ... i/o timeout | Node cannot reach the registry | Network/policy/egress, not a manifest problem |
InvalidImageName | Reference does not parse (uppercase letters, bad characters) | Fix the string |
ErrImageNeverPull | imagePullPolicy: Never and image absent | Load the image or change the policy |
toomanyrequests | Registry rate limit | Mirror or pull through cache registry (see the ECR note below) |
Distinguish this family from CrashLoopBackOff (image pulled fine; the process exits) and CreateContainerConfigError (image pulled fine; a referenced ConfigMap or Secret for env or volumes is missing). Our Pod status guide walks the full state machine.
Serialised vs parallel pulls
By default the kubelet pulls images one at a time per node (serializeImagePulls: true in the KubeletConfiguration). A big image at the front of the queue delays every other Pod scheduled to that node. Administrators can set serializeImagePulls: false and bound concurrency with maxParallelImagePulls (stable since Kubernetes 1.35; must be at least 1, and a value of 2 or more requires serialisation to be off or the kubelet refuses to start). Two rules matter for reasoning about rollouts: pulls are never parallelised within a single Pod (its init and app containers pull in order), and nodes decide independently, so 50 nodes pull the same image 50 times regardless. This is kubelet configuration, not something a CKAD task asks you to change, but it explains why “the Pod on node X started minutes after the others” is sometimes just queueing. The kubelet configuration internals post covers those fields.
Authenticating to a Private Registry
The documentation lists six ways a node can obtain registry credentials. The CKAD cares about two of them, the exam environment uses a third, and the rest are context.
| Method | Who configures it | Scope | When it fits |
|---|---|---|---|
imagePullSecrets on the Pod | Developer | Only Pods that reference the Secret | The recommended default; CKAD task material |
imagePullSecrets on the ServiceAccount | Developer or namespace admin | Every Pod using that ServiceAccount | Avoid repeating the Secret in every manifest; CKAD task material |
| Pre-pulled images on nodes | Cluster admin | All Pods on the node | Exam environments, air-gapped clusters; fragile with autoscaling |
Node-level runtime config (config.json on the node) | Cluster admin | All Pods on the node | Node-wide credentials; merged with Pod-level Secrets |
| Kubelet credential provider plugin | Cluster admin | All Pods, credentials fetched on demand | Cloud registries (ECR, ACR, GCR); short-lived credentials; static Pods |
| Vendor/custom node authentication | Platform | Varies | Managed clusters |
The legacy built-in cloud credential providers for ECR, ACR and GCR were removed in Kubernetes 1.26; clouds now ship exec plugins (see below). If a question describes an EKS node pulling from ECR without any Secret, that is the credential provider path, not magic.
The kubernetes.io/dockerconfigjson Secret, byte by byte
kubectl create secret docker-registry regcred \
--docker-server=registry.example.com \
--docker-username=ci-bot \
--docker-password='s3cr3t' \
[email protected] \
-n team-a
What that creates:
apiVersion: v1
kind: Secret
metadata:
name: regcred
namespace: team-a
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6...
Three facts about the shape. The type is kubernetes.io/dockerconfigjson (the legacy type kubernetes.io/dockercfg is still accepted). The single key must be .dockerconfigjson. Its decoded value is a Docker config.json:
{"auths":{"registry.example.com":{"username":"ci-bot","password":"s3cr3t","email":"[email protected]","auth":"Y2ktYm90OnMzY3IzdA=="}}}
where auth is base64 of username:password. For Docker Hub the server is https://index.docker.io/v1/. To inspect an existing Secret during the exam:
kubectl get secret regcred -n team-a -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
Note the escaped dot in the JSONPath: \.dockerconfigjson, because the key itself begins with a dot. You can also build the Secret from a file you already logged in with:
kubectl create secret generic regcred \
--from-file=.dockerconfigjson=$HOME/.docker/config.json \
--type=kubernetes.io/dockerconfigjson
That fails to be useful if your Docker client uses a credential store: config.json then contains a credsStore entry and no auth values, so use the docker-registry form instead. One Secret can hold multiple registries under auths, and a Pod can list multiple imagePullSecrets; the kubelet tries every credential whose registry matches the image, in order, and falls back to an anonymous pull if none match. Kubernetes is more permissive than Docker when matching auths keys: it allows glob patterns and path prefixes (*.example.com, registry.example.com/team), where each * matches exactly one DNS label, so *.kubernetes.io matches abc.kubernetes.io but not kubernetes.io.
Referencing the Secret from the Pod
apiVersion: v1
kind: Pod
metadata:
name: private-app
namespace: team-a
spec:
imagePullSecrets:
- name: regcred
containers:
- name: app
image: registry.example.com/team/app:1.4.2
imagePullSecrets is a Pod-level field (not per container) and the Secret must be in the same namespace as the Pod. Pods cannot reference Secrets across namespaces, which is why the exam will hand you a Secret in default and a Pod spec in team-a and wait for you to notice. A Pod with containers from two registries lists both Secrets; the kubelet matches each image to the right one.
Attaching the Secret to a ServiceAccount
kubectl patch serviceaccount default -n team-a \
-p '{"imagePullSecrets":[{"name":"regcred"}]}'
or declaratively:
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: team-a
imagePullSecrets:
- name: regcred
The ServiceAccount admission controller then injects imagePullSecrets into every new Pod that uses that ServiceAccount, as if you had written it in the manifest. Two details: it only affects Pods created after the change (existing Pods keep their spec), and if the Pod already lists imagePullSecrets, the admission controller adds the ServiceAccount’s entries rather than replacing them. This is the answer to “make every Pod in the namespace able to pull from the private registry without editing each Deployment”. Our SecurityContext & ServiceAccounts guide covers the other things a ServiceAccount injects.
Kubelet credential providers (awareness)
A credential provider is an exec plugin the kubelet runs to obtain short-lived registry credentials on demand, communicating over stdin/stdout. It has been stable since Kubernetes 1.26. The kubelet is pointed at it with two flags, --image-credential-provider-config and --image-credential-provider-bin-dir, and the config is a kubelet.config.k8s.io/v1 CredentialProviderConfig whose providers entries name the binary, list matchImages patterns (globs allowed in the domain only, one label per *, path is a prefix match, port must match exactly), and set a required defaultCacheDuration. Since 1.33, with the KubeletServiceAccountTokenForCredentialProviders feature gate, a provider can receive the Pod’s ServiceAccount token so workloads pull with their own identity rather than a node role. You will not configure this on the CKAD, but you should recognise it as the reason cloud clusters pull from ECR, ACR or GCR with no Secret anywhere in the manifest. The registry-side mechanics for ECR (pull through cache, replication, lifecycle policies) are in our Amazon ECR internals post.
Credential Verification for Images Already on the Node
Here is a subtle security gap that IfNotPresent opens. Tenant A pulls a private image using its Secret; the image is now cached on the node. Tenant B schedules a Pod with the same image reference and no credentials. With IfNotPresent, the kubelet sees the image locally and starts the container. Tenant B just ran an image it was never authorised to pull. The classic mitigation was the AlwaysPullImages admission controller, at the cost of a registry round trip for every start.
Kubernetes now has a targeted fix: the KubeletEnsureSecretPulledImages feature gate (alpha in 1.33, beta and on by default since 1.35). When enabled, the kubelet records which credentials successfully pulled each image, and for any image that required credentials it verifies the requesting Pod’s credentials before reusing the cached image. Credentials that previously succeeded are validated locally; new credentials force a re-pull from the registry. The behaviour is tuned with imagePullCredentialsVerificationPolicy in the KubeletConfiguration:
| Policy | Meaning |
|---|---|
NeverVerify | Same as the gate being off; cached images are never verified |
NeverVerifyPreloadedImages | Images pulled outside the kubelet (pre-loaded) are trusted; everything the kubelet pulled is verified. Default |
NeverVerifyAllowlistedImages | Only pre-loaded images listed in preloadedImagesVerificationAllowlist are trusted |
AlwaysVerify | Every image is verified before use |
For the CKAD, the takeaway is conceptual: on a modern cluster, “the image is already on the node so I don’t need the Secret” is no longer reliably true, and the correct manifest always carries the credentials it needs.
Image Volumes: Mounting an OCI Artifact Instead of Running It
The newest image-related feature a CKAD candidate should know is the image volume type. It lets a Pod mount the contents of a container image or OCI artifact as a read-only volume, which is how you ship ML model weights, configuration bundles, static assets or language runtimes separately from the application image. The ImageVolume feature gate went alpha in 1.31, beta (off) in 1.33, beta on by default in 1.35 and stable in 1.36.
apiVersion: v1
kind: Pod
metadata:
name: model-server
spec:
containers:
- name: server
image: registry.example.com/team/inference:2.1
volumeMounts:
- name: weights
mountPath: /models
readOnly: true
volumes:
- name: weights
image:
reference: registry.example.com/team/model-weights:v7
pullPolicy: IfNotPresent
Rules worth memorising:
referenceis handled exactly likecontainers[*].image, and pull Secrets are assembled the same way, soimagePullSecretson the Pod cover image volumes too.pullPolicyhas the same three values, with one difference in the default:Alwaysonly when the tag is:latest, otherwiseIfNotPresent.- A failed resolve or pull blocks the containers from starting; with
AlwaysorNeverfailures the Pod goes toFailed. - The mount is read-only, in a single directory.
subPathandsubPathExprwork from 1.33.fsGroupChangePolicyhas no effect, and theAlwaysPullImagesadmission controller applies to image volumes as it does to container images. - The volume is re-resolved only when the Pod is deleted and recreated, so a moved tag is not picked up by a container restart.
If a task says “the application must read files from a separate OCI image at /data without copying them into the application image”, this is the volume type, not an init container that copies files into an emptyDir (though that older pattern still works and is covered in our volumes guide).
command and args: What Runs Once the Image Is Present
The image’s ENTRYPOINT and CMD are defaults; the Pod spec’s command and args override them. The resolution matrix is the single most tested fact about this topic:
Image ENTRYPOINT | Image CMD | Pod command | Pod args | What runs |
|---|---|---|---|---|
[/ep] | [c1] | — | — | /ep c1 (image defaults) |
[/ep] | [c1] | [/cmd] | — | /cmd (CMD is dropped too) |
[/ep] | [c1] | — | [a1] | /ep a1 (ENTRYPOINT kept, CMD replaced) |
[/ep] | [c1] | [/cmd] | [a1] | /cmd a1 (image defaults ignored) |
Two rules from that table: setting command alone discards the image’s CMD, which surprises people who expected the default arguments to survive; and setting args alone is the right way to pass different parameters to the image’s own entrypoint. command and args are immutable after the Pod is created: change them in a Deployment template (which creates new Pods), never by editing a running Pod.
containers:
- name: worker
image: registry.example.com/team/worker:3.0
command: ["/bin/sh", "-c"]
args:
- |
echo "starting on $(HOSTNAME) port $(PORT)"
exec /app/worker --port=$(PORT) --level="$LOG_LEVEL"
env:
- name: PORT
value: "8080"
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: worker-config
key: level
Pipes, globs, && and shell variables ($LOG_LEVEL) need a shell, hence /bin/sh -c. Without a shell, command: ["echo", "hello | grep h"] passes the pipe as a literal argument.
$(VAR) expansion: Kubernetes, not the shell
Kubernetes performs its own variable expansion in command, args and env[].value using the $(VAR_NAME) syntax, before the container starts and independently of any shell. The rules:
- Only variables defined in the container’s
env(from literal values, ConfigMaps, Secrets or the Downward API) are candidates. - Order matters: a variable can only reference variables defined earlier in the
envlist. Reference one defined later and the text is left as the literal$(VAR_NAME). - An undefined reference is also left literal and does not prevent the container from starting.
$$(VAR_NAME)escapes the reference and is never expanded, whether or not the variable exists.
The documentation’s demo makes this concrete. With env ordered SERVICE_PORT, SERVICE_IP, UNCHANGED_REFERENCE=$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT), PROTOCOL=https, SERVICE_ADDRESS=$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT), ESCAPED_REFERENCE=$$(PROTOCOL)://..., the container logs:
UNCHANGED_REFERENCE=$(PROTOCOL)://172.17.0.1:80
SERVICE_ADDRESS=https://172.17.0.1:80
ESCAPED_REFERENCE=$(PROTOCOL)://172.17.0.1:80
UNCHANGED_REFERENCE failed because PROTOCOL was defined below it; SERVICE_ADDRESS succeeded because by then PROTOCOL existed. Rule 3 also explains a pattern you have typed many times without thinking: args: ["-c", "echo $(date)"]. Kubernetes looks for an env var named date, finds none, leaves $(date) literal, and the shell then performs command substitution. If you ever define an env var called date, that script changes meaning. Conversely, $PORT (no parentheses) is not touched by Kubernetes and only works inside a shell.
The expansion also reaches valueFrom-sourced variables: a PORT from a ConfigMap can be used in a later args: ["--port=$(PORT)"] with no shell at all, which is the fastest exam-safe way to parameterise an entrypoint.
Diagnosis Flow for the Exam
kubectl get pod app -o wide # STATUS column: ErrImagePull / ImagePullBackOff / ErrImageNeverPull
kubectl describe pod app | sed -n '/Events:/,$p' # the real message
kubectl get pod app -o jsonpath='{.spec.containers[*].image}{"\n"}{.spec.containers[*].imagePullPolicy}{"\n"}{.spec.imagePullSecrets}{"\n"}'
kubectl get secret -n $(kubectl get pod app -o jsonpath='{.metadata.namespace}') # is the Secret here?
kubectl get sa default -o jsonpath='{.imagePullSecrets}{"\n"}'
Then act:
| Symptom | Decision |
|---|---|
Image exists locally, tag is latest or absent | Add a real tag or set imagePullPolicy: IfNotPresent |
manifest unknown | Typo in tag/path; kubectl set image deploy/app app=registry/app:1.4.2 |
unauthorized, Pod has no imagePullSecrets | kubectl create secret docker-registry in the Pod’s namespace, add to Pod or ServiceAccount, recreate Pod |
FailedToRetrieveImagePullSecret | Secret name/namespace mismatch |
Fixed but still ImagePullBackOff | Back-off timer (up to 5 min); delete the Pod or kubectl rollout restart |
| Container runs the wrong thing | Check the command/args matrix; remember command drops CMD |
Argument prints $(VAR) literally | Variable undefined, defined later in env, or escaped with $$ |
| Container starts but files from the artifact image are missing | Image volume reference/pullPolicy, or missing subPath (1.33+) |
Hands-On Lab (15 Minutes)
kubectl run t1 --image=busybox --command -- sleep 3600, then read back.spec.containers[0].imagePullPolicy. It isAlwaysbecause no tag was given.kubectl run t2 --image=busybox:1.36 --command -- sleep 3600; the policy isIfNotPresent.kubectl set image pod/t2 t2=busybox:latest(allowed: image is mutable on a Pod) and read the policy again. StillIfNotPresent: it froze at creation.kubectl run t3 --image=nginx:9.9.9and watchkubectl get pod t3 -wcycleErrImagePull→ImagePullBackOff; time the retries indescribe.- Create
regcredwithkubectl create secret docker-registryand decode.dockerconfigjsonwith the escaped JSONPath. - Patch the
defaultServiceAccount with the Secret, create a new Pod, and confirmimagePullSecretsappeared in its spec without you writing it. - Write a Pod whose
argsreference an env var defined after it; read the logs; reorder; read again. - If your cluster is 1.36 or newer, mount
registry.k8s.io/conformance:<your version>as animagevolume andlsthe mount.
The point of the lab is speed: on the exam you want the imagePullPolicy table, the Secret shape, the same-namespace rule and the $(VAR) ordering rule to be reflexes, not things you look up. The CKAD Certification Ready Mock Exam Bundle runs five timed, performance-based exams in a real cluster, where a stalled ImagePullBackOff costs you the same minutes it costs on test day; practising the diagnosis flow there, with graded results, is the fastest way to make sure the image pull is the part of the task you never think about again. Pair it with our CKAD study plan and the kubectl cheat sheet for the imperative forms above.
Common Mistakes
- Relying on the default pull policy for a local-only image.
latestor no tag meansAlways; no registry meansImagePullBackOff. - Changing the tag to
latestand expecting pulls. The stored policy does not update; setAlwaysexplicitly. - Creating the registry Secret in the wrong namespace.
imagePullSecretscan only reference Secrets in the Pod’s namespace. - Attaching the Secret to the ServiceAccount and expecting running Pods to pick it up. Only new Pods are mutated.
- Setting
commandand wondering where the image’s arguments went.commandreplacesENTRYPOINTand dropsCMD. - Referencing an env var before it is defined.
$(VAR)resolves top-down through theenvlist. - Assuming a cached private image is free for every tenant.
KubeletEnsureSecretPulledImages(beta on since 1.35) verifies credentials;AlwaysPullImagesis the older mitigation. - Deleting and recreating Pods to “fix” the back-off without fixing the cause. The timer resets; the error returns.
Frequently Asked Questions
What is the default imagePullPolicy in Kubernetes?
It depends on the image reference when the Pod is created. If the tag is latest or there is no tag, the default is Always. If there is any other tag or a digest, the default is IfNotPresent. The computed value is stored in the Pod and does not change if you later edit the image tag.
What is the difference between ErrImagePull and ImagePullBackOff?
ErrImagePull is the reason shown immediately after a pull attempt fails. The kubelet then waits before retrying, and during that wait the reason is ImagePullBackOff. The delay grows exponentially up to a cap of 300 seconds (5 minutes), and the kubelet retries indefinitely. The underlying cause is in the Pod’s events, not in the reason string.
Does imagePullSecrets have to be in the same namespace as the Pod?
Yes. A Pod can only reference image pull Secrets in its own namespace, and the Secret must be of type kubernetes.io/dockerconfigjson (or the legacy kubernetes.io/dockercfg). If the name or namespace is wrong you will see a FailedToRetrieveImagePullSecret event.
How do I make every Pod in a namespace use a registry Secret?
Add the Secret to the ServiceAccount the Pods use, for example kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"regcred"}]}'. The ServiceAccount admission controller injects imagePullSecrets into every Pod created afterwards. Existing Pods are not changed.
If I set command in a Pod, is the image’s CMD still used as arguments?
No. Setting command replaces the image’s ENTRYPOINT and discards its CMD. If you want the image’s entrypoint with different arguments, set only args. Setting both replaces both.
Why does $(VAR) appear literally in my container’s arguments?
Either the variable is not defined in the container’s env, it is defined later in the env list than the place that references it, or you escaped it with $$(VAR). Kubernetes expands references top-down and leaves unresolved ones as literal text without failing the container.
What is an image volume in Kubernetes?
An image volume mounts the contents of a container image or OCI artifact into a Pod as a read-only directory, using the same pull Secrets and a pullPolicy field like a container image. The ImageVolume feature became stable in Kubernetes 1.36; subPath support arrived in 1.33. It is the native way to ship model weights or configuration bundles separately from the application image.
Can a Pod reuse a private image another Pod already pulled onto the node?
Historically yes with IfNotPresent, which was a security gap closed by the AlwaysPullImages admission controller. Since Kubernetes 1.35 the KubeletEnsureSecretPulledImages feature (beta, on by default) makes the kubelet verify that the new Pod’s credentials are valid for the cached image before reusing it, with the default policy NeverVerifyPreloadedImages.
Conclusion
The pull is the part of a CKAD task you want to get right without thinking, and that only happens once you know the rules. The default imagePullPolicy is Always for latest or no tag and IfNotPresent otherwise, and it freezes when the Pod is created. A failed pull cycles ErrImagePull → ImagePullBackOff with a back-off that caps at five minutes and never gives up. Private registries are handled with a kubernetes.io/dockerconfigjson Secret referenced from the Pod or injected by its ServiceAccount, always in the Pod’s own namespace, while cloud clusters use kubelet credential providers you never see. Modern kubelets verify credentials before reusing cached private images, image volumes mount OCI artifacts read-only, and once the image is present command and args resolve against ENTRYPOINT and CMD by a four-row matrix with $(VAR) expansion that runs top-down through env. Learn those rules once and the ten minutes you used to lose to ImagePullBackOff become ten minutes you spend on the task itself.