Every KubeAstronaut study guide tells you that the API server is the front door of the cluster, that controllers “watch” objects and reconcile desired with actual state, and that kubectl is an HTTP client. Our Kubernetes object model guide and architecture fundamentals guide say exactly that, because they are about the shape of the system. Those sentences are enough for the KCNA question that asks “which component do controllers talk to”.
They are not enough for the questions, and the real-world incidents, that live one level down. Why does kubectl edit sometimes fail with “the object has been modified; please apply your changes to the latest version” while kubectl apply of the same change does not? What does a controller mean when its log says too old resource version: 41203 (41987), and why is the correct response to throw away its cache? Why does kubectl get pods -A on a large cluster issue several requests, and what happens if one of the later ones arrives six minutes after the first? Why does kubectl rollout status print “Waiting for deployment spec update to be observed”? What is the difference between the uid, the resourceVersion and the generation that every object carries? This is the deep dive those questions deserve.
It is written for the CKA and CKAD candidate who needs to debug these symptoms under time pressure, and for the KCNA candidate who needs the vocabulary. Everything below is checked against the Kubernetes v1.37 documentation and the v1.37 feature-gate tables. Read it once for the model, then use the symptom drill near the end to test whether it stuck.
Where This Sits in the Five Exams
The KubeAstronaut title requires CKA, CKAD, CKS, KCNA and KCSA, and this material surfaces in four of them from different angles:
| Exam | Why it appears |
|---|---|
| KCNA | Kubernetes Fundamentals (46%) and Architecture: what the API server, etcd, controllers and kubectl do; what “watch” means; what a controller loop is |
| CKA | Troubleshooting (30%) and Cluster Architecture (25%): reading controller logs, understanding 409/410/429 errors, API server flags, etcd history |
| CKAD | Application Observability and Maintenance: kubectl get -w, kubectl wait, rollout status, debugging why a change “did not apply” |
| KCSA | API server exposure and denial of service: expensive list requests and API Priority and Fairness, covered in our DoS and APF guide and only cross-referenced here |
The authentication, authorisation and admission chain that every request passes through is owned by the authorization chain guide and is not retaught. This post starts after admission: how a request reads and writes storage, and how clients stay in sync with it.
Every kubectl Command Is One or More HTTP Requests
The Kubernetes API is a resource-based HTTP interface. A resource type is the name in the URL (pods, deployments); a kind is the schema of the object returned (Pod, Deployment). URLs follow two shapes:
Cluster-scoped: /apis/GROUP/VERSION/RESOURCETYPE[/NAME]
Namespace-scoped: /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE[/NAME]
Core group: /api/v1/... (no GROUP segment, /api instead of /apis)The API verbs are get, list, watch, create, update, patch, delete, deletecollection and proxy; a watch is a GET on a collection with ?watch=1. Only list, watch and deletecollection operate on more than one object. These are the same verbs you grant in RBAC rules, which is why a Role that allows get but not list lets kubectl get pod web succeed and kubectl get pods fail with a 403.
You can watch the traffic yourself. kubectl logs the HTTP exchange at increasing verbosity:
| Flag | Shows |
|---|---|
-v=6 | The requested URL and response code for each call |
-v=7 | HTTP request headers |
-v=8 | Request and response bodies, truncated |
-v=9 | Full bodies, plus an equivalent curl command |
kubectl get pods -n kube-system -v=6 2>&1 | grep -E 'GET|Response'
# I1104 ... GET https://10.0.0.10:6443/api/v1/namespaces/kube-system/pods?limit=500 200 OK in 9 msTwo details in that one line are exam material. First, limit=500: kubectl get paginates by default with --chunk-size=500 (set --chunk-size=0 to disable). Second, the request asked for a Table: at -v=7 you will see Accept: application/json;as=Table;v=v1;g=meta.k8s.io, application/json, which tells the server to render the columns you see in the terminal server-side (--server-print=false turns that off and makes kubectl format locally). -o json and -o yaml request the real object instead, which is why those outputs are larger and why the output formatting guide tells you to script against them rather than against table text.
Before any of that, kubectl must know which resources exist. It calls the discovery endpoints (/api, /apis, and since 1.30 the aggregated discovery document) and caches the result under ~/.kube/cache/discovery/ for a few hours, which is why a freshly installed CRD sometimes needs a moment, or a cache wipe, before kubectl get <newkind> works. kubectl api-resources --verbs=list,watch reads from the same cache and is the quickest way to see which resource types support watching.
You can also skip kubectl’s object handling entirely and issue a raw request through your kubeconfig credentials:
kubectl get --raw '/api/v1/namespaces/default/pods?limit=1' | jq '.metadata'
# { "resourceVersion": "812340", "continue": "eyJ2IjoibWV0YS5r...", "remainingItemCount": 41 }That metadata block is the whole subject of the next few sections.
Three Identities Every Object Carries
Open any object with -o yaml and the first thing under metadata is a cluster of fields that look similar and mean different things.
| Field | Identifies | Changes when | Set by |
|---|---|---|---|
name + namespace (+ group and resource) | The logical object | Never for a given object | You |
uid | This instance of the object | Never; a deleted and recreated object gets a new uid | API server on create |
resourceVersion | The storage version at which this instance was last written | On every write, including status and metadata updates | API server, from etcd |
generation | The version of the object’s desired state | Only when spec changes (for resources that track it) | API server |
status.observedGeneration | The last generation the controller has acted on | When the controller finishes processing a spec change | The controller |
uid is what owner references point at, which is why a ReplicaSet re-created with the same name does not adopt the Pods of its predecessor, and why kubectl delete can take a precondition on uid to avoid deleting a same-named replacement. The garbage-collection side of this is in the finalizers and owner references guide.
resourceVersion is the one most people misunderstand. The documentation defines it as a string that “identifies the server’s internal version of an object”, which “must be passed unmodified back to the server”. For everything served by kube-apiserver it is derived from the etcd revision at which the write happened, so it is a single cluster-wide counter: a Pod written at revision 812340 and a ConfigMap written at 812341 have consecutive resource versions even though they are unrelated. Since 1.35 the docs make orderability a conformance requirement: for the same resource type, resource versions from kube-apiserver compare as monotonically increasing integers. For resources served by an extension API server the only safe operation is equality. In practice: never parse it in your own code, but understand that “newer” means “larger” when you are reading logs.
Two places you see it: on an object (.metadata.resourceVersion, the version it was last modified at) and on a list response (.metadata.resourceVersion, the version at which the collection was constructed). The list one is what you hand to a watch.
generation answers a different question: has the desired state changed? The API server increments it when spec changes and leaves it alone when only status or most metadata changes (for resources that declare a status subresource, the status endpoint cannot bump it). Controllers then publish status.observedGeneration. When observedGeneration < generation, the controller has not yet seen your edit. That is precisely what kubectl rollout status is waiting for when it prints “Waiting for deployment spec update to be observed”, and it is the first check in the Deployment rollout internals guide. Pods gained the same tracking when PodObservedGenerationTracking went GA in 1.35, which the in-place resize guide uses to tell whether the kubelet has acted on a resize.
Optimistic Concurrency and the 409 Conflict
Kubernetes has no locks. Instead, every update (PUT) and most patch requests can carry the resourceVersion the client last saw, and the API server treats it as a precondition: if the stored object has moved on, the write is rejected with HTTP 409 and this message:
Error from server (Conflict): Operation cannot be fulfilled on deployments.apps "web":
the object has been modified; please apply your changes to the latest version and try againWhich commands hit this depends entirely on whether they send a resourceVersion:
| Command | Sends resourceVersion? | Conflict behaviour |
|---|---|---|
kubectl edit | Yes, the one in the object you opened | Conflicts if anything (a controller writing status, HPA scaling, another admin) touched the object while your editor was open. kubectl edit reopens the editor with the error so you can retry |
kubectl replace -f | Only if your file contains one | With it: conflicts on mismatch. Without it: unconditional overwrite, which is why replace “works” and quietly discards someone else’s change |
kubectl apply -f | No (client-side apply sends a PATCH without a precondition) | Rarely conflicts. Server-side apply can return 409 for a different reason, a field-manager conflict, which is owned by the apply/create/replace guide |
kubectl scale, kubectl set image, kubectl patch | No by default | Patches without a precondition merge onto the latest object |
| Controllers using client-go | Yes, from their cache | They call retry.RetryOnConflict: re-read, re-apply the change, re-submit |
The exam-relevant conclusions: a 409 is not an error in the cluster, it is the concurrency control working; the fix is “get the latest version and reapply your change”, never --force; and a manifest that contains a stale resourceVersion (because someone saved -o yaml output into Git) will fail kubectl replace and kubectl create until the field is removed. The same stale-field problem is why the CRD and Operators guide tells you to strip uid, resourceVersion, creationTimestamp and managedFields before reusing an exported object.
You can reproduce the conflict in a few seconds, which is a useful way to make it stick:
kubectl create deployment web --image=nginx
kubectl get deploy web -o json > a.json # resourceVersion N
kubectl scale deploy web --replicas=2 # object is now at N+something
kubectl replace -f a.json # Conflict: a.json still says N
jq 'del(.metadata.resourceVersion)' a.json | kubectl replace -f - # unconditional overwrite succeedsget and list: What resourceVersion Means on a Read
Clients can also pass resourceVersion on a get or list, and here it is not a precondition but a consistency request. The semantics table from the API concepts page is worth memorising because the defaults are the opposite of what most people assume:
resourceVersion on the request | Semantics | Served from |
|---|---|---|
| unset | Most Recent: the data must be consistent, i.e. a quorum read from etcd | etcd (in 1.37, the watch cache after it confirms via etcd progress notifications that it is caught up) |
"0" | Any: the newest version the server has is preferred but staleness is tolerated | Always the watch cache |
| a specific value | Not older than that version (list requires resourceVersionMatch=NotOlderThan) | Always the watch cache |
a specific value + resourceVersionMatch=Exact | Exact that version | A cache snapshot if available (ListFromCacheSnapshot, beta on by default since 1.34), otherwise etcd; 410 Gone if unavailable |
a continue token | Continuation of a paginated list at the version the first page used | Snapshot or etcd |
Three consequences for how you read the cluster:
kubectl getis a quorum read.kubectlsends noresourceVersion, so it asks for Most Recent. That is whykubectl getimmediately reflects a write you just made, and also why a storm ofkubectl getloops in a CI job puts load on etcd. The 1.37 note that most-recent reads can be served from the watch cache (with etcd 3.4.31+ / 3.5.13+) is a performance change, not a semantic one: the server still guarantees the data is at least as new as etcd’s latest revision, it just proves it with a progress notification instead of a full quorum read.- Informers start with
"0". client-go’s reflector lists withresourceVersion=0so that the initial list of thousands of objects comes from the API server’s memory rather than etcd. The docs state the reason: “Unless you have strong consistency requirements, usingresourceVersionMatch=NotOlderThanand a knownresourceVersionis preferable since it can achieve better performance and scalability”. - Setting a version on a list without
resourceVersionMatchis legacy. The docs say you “should always set theresourceVersionMatchparameter when settingresourceVersionon a list request”, and settingresourceVersionMatchwithout a version is invalid.
Ask for a version the server has never seen, for instance a number from a different cluster or a typo, and the API server waits briefly, then returns 504 Gateway Timeout with the message Too large resource version, or a Retry-After header. A watch for an unknown version may block until the request timeout.
Pagination: limit, continue and the 5-Minute Clock
Listing every Pod in a 5,000-node cluster in one response would be an enormous allocation on the server and the client. Chunked listing has been stable since 1.29: the client sends limit=N, the server returns up to N items plus a metadata.continue token, and the client repeats with continue=<token> until the token is empty. metadata.remainingItemCount estimates what is left, but only for unfiltered lists (it is omitted when you use label or field selectors, and on the final page).
The important guarantee is consistency across pages: the resourceVersion is fixed by the first page and every later page is served at that same version, so the pages together form a snapshot. Changes after that version are not visible until you list again. That is what lets a client page through a large collection and then start a watch from the snapshot’s version “without missing any updates”.
The cost of that guarantee is the clock. The continue token encodes the snapshot’s resource version, and the server can only serve it while that version is still available. The docs put the default at about 5 minutes, which lines up with kube-apiserver --etcd-compaction-interval=5m0s: every five minutes the API server asks etcd to compact history older than the current revision. After compaction, a request with an old token gets HTTP 410 Gone with reason Expired:
The provided continue parameter is too old to display a consistent list result.
You can start a new list without the continue parameter, or use the continue token
in this response to retrieve the remainder of the results. Continuing with the provided
token results in an inconsistent list - objects that were created, modified, or deleted
between the time the first chunk was returned and now may show up in the list.Notice the choice the server offers: start over for a consistent view, or take the new token it hands you and finish with an inconsistent one. kubectl takes the second option silently, which is why a slow kubectl get pods -A on a huge cluster can, in rare cases, show an object twice or miss one, and why scripts that need an exact census should list without --chunk-size or re-list on 410.
A limit also interacts with resourceVersion in two ways worth knowing: limit with a specific version and no resourceVersionMatch is treated as Exact, and limit with continue plus any explicit resourceVersion is rejected with 400 because the token already carries the version.
Watch: Change Notifications Over a Long-Lived GET
A watch is a GET on a collection with ?watch=1, returned with Transfer-Encoding: chunked, on which the server streams one JSON event per line for as long as the connection lives. Each event has a type and an object:
type | Meaning | object contains |
|---|---|---|
ADDED | A new object, or a synthetic event describing an existing object at watch start | The full object |
MODIFIED | Any write to the object, including status-only writes | The full new object |
DELETED | The object is gone | The object’s last state |
BOOKMARK | ”Everything up to this version has been sent” | Only metadata.resourceVersion |
ERROR | The watch is being terminated | A Status (for example code 410, reason Expired) |
There is no per-field watch and no MODIFIED diff; you receive whole objects and compare them yourself. Every write produces an event, so a Deployment whose controller updates status three times generates three MODIFIED events with identical spec.
The list-then-watch pattern
The canonical client sequence from the API concepts page is: list the collection, remember the list’s metadata.resourceVersion (say 10245), then watch from it:
GET /api/v1/namespaces/test/pods
200 OK {"kind":"PodList","metadata":{"resourceVersion":"10245"},"items":[...]}
GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
200 OK Transfer-Encoding: chunked
{"type":"ADDED", "object":{"kind":"Pod","metadata":{"resourceVersion":"10596",...}}}
{"type":"MODIFIED", "object":{"kind":"Pod","metadata":{"resourceVersion":"11020",...}}}Because the watch starts at exactly the version the list captured, no change can fall between them. This is the mechanism behind kubectl get pods -w, behind kubectl wait --for=condition=Ready pod/web (and --for=delete, --for=jsonpath='{.status.phase}'=Running, and since 1.31 --for=create), and behind every controller in the cluster.
What resourceVersion means on a watch
resourceVersion on the watch | Semantics | Synthetic ADDED events first? |
|---|---|---|
| unset | Get State and Start at Most Recent: consistent starting point (quorum read) | Yes, one per existing object |
"0" | Get State and Start at Any: may start from a stale cached state, which can “rewind” the client | Yes |
| a specific value | Start at Exact: the client already has state at that version | No |
kubectl get pods -w uses the first form: it lists, prints the table, then watches from the list’s version. kubectl get pods -w --watch-only skips the initial list and prints only changes; --output-watch-events prints the event type alongside each row, which is the fastest way to see ADDED/MODIFIED/DELETED during a lab.
Watches end, by design
A watch connection is not permanent. kube-apiserver --min-request-timeout (default 1800 seconds) is “currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load”. So every watch is closed somewhere between 30 and 60 minutes after it opened, and the client is expected to reconnect from its last seen resourceVersion. Seeing watches close every half hour or so in logs is normal. Non-watch requests use --request-timeout (default 1m0s), which is why a kubectl get that hangs for exactly a minute before erroring is telling you the server accepted the request but could not answer it, for example because etcd is slow.
410 Gone, “too old resource version”, and Bookmarks
The API server keeps history for only a short window. The docs: “Clusters using etcd 3 preserve changes in the last 5 minutes by default”, the compaction interval again. The API server’s in-memory watch cache (--watch-cache=true) holds a bounded buffer of recent events per resource type on top of that, and the per-resource --watch-cache-sizes flag now only accepts zero, meaning “disable the cache for this resource”.
If a client reconnects a watch from a resourceVersion older than what the server still holds, it gets an ERROR event, or a 410 on the initial request, carrying a Status with reason Expired and a message like too old resource version: 41203 (41987): the version you asked for, then the oldest version still available. The documented client contract is unambiguous: recognise the 410, clear the local cache, perform a new list, and start a new watch from the resourceVersion the list returned. A client that instead retries the same watch forever is broken, and a controller log full of too old resource version followed by a relist is a controller behaving correctly after a network blip or an API server restart.
Why would a client fall out of the window? Because it was disconnected for longer than five minutes, or because the resource churns so fast that five minutes of history is more than the cache keeps. Bookmarks shrink the second risk. A client that sends allowWatchBookmarks=true may receive periodic BOOKMARK events whose object holds only a resourceVersion. A bookmark says “all changes up to here have already been sent”, so even if no object the client cares about has changed in an hour (for instance a watch with a label selector that matches nothing), the client’s “last seen version” keeps advancing and its reconnect will land inside the window. The docs are careful about the guarantee: bookmarks are not promised at any interval, and the server may send none even when asked. client-go requests them by default.
Streaming Lists: Replacing the Big List With a Watch
The initial list of a large collection has always been the expensive step: the server builds the whole response in memory, and a thousand controllers restarting after an upgrade can all do it at once. Streaming lists change the first step of list-then-watch into a watch with sendInitialEvents=true:
GET /api/v1/namespaces/test/pods?watch=1&sendInitialEvents=true&allowWatchBookmarks=true&resourceVersion=&resourceVersionMatch=NotOlderThanThe server streams a synthetic ADDED event for every existing object, then a BOOKMARK whose object carries the annotation k8s.io/initial-events-end: "true" and the resource version at which the state is complete, then continues as a normal watch. The request must use resourceVersionMatch=NotOlderThan; an empty resourceVersion asks for a consistent (most recent) starting state, while a specific value asks for state at least that new.
Feature-gate history here is a classic version-skew question: WatchList on the API server was alpha from 1.27, beta and on in 1.32, off again in 1.33 while an encoding problem was fixed, and beta and on by default since 1.34. The client side, WatchListClient in client-go, has been beta and on by default since 1.35, so current controller managers and informers use streaming by default against a 1.34+ server and fall back to list+watch against older ones. A related server-side improvement, StreamingCollectionEncodingToJSON/ToProtobuf (GA in 1.34), lets ordinary list responses be encoded item by item instead of as one giant buffer. Together these are why “API server OOM during mass controller restart” is a much rarer incident than it was in the 1.2x series.
An alpha feature in 1.36/1.37, ShardedListAndWatch, goes further and lets a horizontally scaled controller watch only a hash range of objects (shardRange(object.metadata.uid, ...)). It is disabled by default and not exam material yet, but it shows where this API is going.
Informers: How list+watch Becomes a Controller
The KCNA workloads guide explains what each controller does. This is how any of them does it, in client-go terms, because the exam troubleshooting questions make more sense once you can picture it:
- Reflector. Performs the list (or streaming list) and then the watch for one resource type, handles 410 by relisting, reconnects on timeout from the last seen version, and asks for bookmarks.
- DeltaFIFO. A queue of
Added/Updated/Deleteddeltas fed by the reflector. - Indexer (the local cache). An in-memory store of the latest version of every object, indexed by namespace/name and by any custom indexes. Controllers read from this cache, never from the API server, which is why a controller can be briefly stale and why reads of the cache cost etcd nothing.
- Event handlers.
OnAdd/OnUpdate/OnDeletecallbacks that do almost no work except pushing a key such asdefault/webonto a workqueue. The queue deduplicates (ten rapid updates to one Deployment become one reconcile) and rate-limits with exponential backoff on failure. - Reconcile. A worker pops a key, reads the current object from the cache, computes what should exist, and writes through the API server with
resourceVersionpreconditions andRetryOnConflict. A periodic resync re-queues every cached key even if nothing changed, so a reconcile that failed silently is eventually retried.
The design is level-triggered: the reconcile looks at the current state and converges, it does not replay a history of events. That property is what makes 410-and-relist safe (the controller just sees the current world again), what makes bookmarks optional, and what the controller concepts page means when it says a controller tries to move “the current cluster state closer to the desired state” and that the cluster may never be stable. For the exam: a controller that “missed an event” is not a thing you fix by replaying events; you fix it by letting it resync or restarting it so it relists.
Built-in controllers run inside kube-controller-manager and elect one active instance through a Lease object in kube-system; the scheduler does the same, and its own list/watch-driven cycle is the subject of the scheduling framework guide. Every one of them, plus the kubelet watching Pods bound to its node, is an informer. That is why an API server that cannot serve watches takes the whole cluster’s control loops with it even though running Pods keep running.
The kube-apiserver Flags That Shape All of This
| Flag | Default | What it governs |
|---|---|---|
--etcd-compaction-interval | 5m0s | How often the API server asks etcd to compact history; sets the practical lifetime of continue tokens and old watch versions. 0 disables apiserver-driven compaction |
--min-request-timeout | 1800 | Lower bound (seconds) of the randomised watch connection lifetime |
--request-timeout | 1m0s | Timeout for ordinary requests |
--watch-cache | true | Enables the in-memory per-resource cache that serves “Any”/“Not older than” reads and fans out watch events |
--watch-cache-sizes | unset | resource#0 disables the cache for that resource; non-zero values are all equivalent |
--event-ttl | 1h0m0s | How long Event objects are kept; unrelated to watch history despite the name |
--max-requests-inflight / --max-mutating-requests-inflight | 400 / 200 | Summed into the total concurrency limit when --enable-priority-and-fairness (default true) is on |
--storage-media-type | application/vnd.kubernetes.protobuf | How objects are encoded in etcd; etcdctl get output is binary for this reason |
Two of these connect to other exam topics. Because --storage-media-type is protobuf, you cannot read objects out of etcd with etcdctl and jq; the etcd backup and restore guide covers what you can do there. And because every list is a potentially large read, API Priority and Fairness (which replaced simple in-flight limits) estimates the cost of list requests, with SizeBasedListCostEstimate beta and on since 1.34, and will queue or reject (429) a flood of expensive lists before they starve kubelet heartbeats. The DoS and APF guide owns FlowSchemas and PriorityLevelConfigurations.
One more 429 source is new enough to catch people: while a freshly started API server’s watch cache is still initialising, watches and un-paginated or filtered lists receive 429 Too Many Requests with a Retry-After header instead of being served from etcd (ResilientWatchCacheInitialization GA 1.34, WatchCacheInitializationPostStartHook GA 1.37, which also delays readiness until the cache is warm). A burst of 429s in the first seconds after a control-plane restart is therefore expected, not an outage.
Symptom-to-Cause Drill
Cover the right-hand column and diagnose.
| # | Symptom | Cause and action |
|---|---|---|
| 1 | kubectl edit fails: “the object has been modified; please apply your changes to the latest version” | 409 optimistic-concurrency conflict; something wrote the object while the editor was open. Re-run edit and reapply |
| 2 | kubectl replace -f saved.yaml fails with Conflict, but the YAML is “correct” | The file carries a stale resourceVersion. Delete the field (or re-export) |
| 3 | kubectl create -f exported.yaml fails: resourceVersion should not be set on objects to be created | Same stale field; strip uid, resourceVersion, creationTimestamp, managedFields |
| 4 | Controller log: watch of *v1.Pod ended with: too old resource version: 41203 (41987) then Listing and watching *v1.Pod | Watch fell outside the history window; the reflector relisted. Normal after disconnects or apiserver restarts |
| 5 | The same message every few seconds, forever | Clock skew is not involved; the resource churns faster than the cache window, or the client is custom code that does not relist. Check --etcd-compaction-interval and the client’s 410 handling |
| 6 | Large kubectl get pods -A prints “The provided continue parameter is too old” at -v=6 but completes | Pagination token expired after about five minutes; kubectl continued inconsistently. Use --chunk-size=0 for a strict census |
| 7 | kubectl rollout status sits on “Waiting for deployment spec update to be observed” | status.observedGeneration < metadata.generation; the Deployment controller has not processed the spec. Check kube-controller-manager health and leader Lease |
| 8 | Many 429 Too Many Requests right after the control plane restarted, then silence | Watch cache initialising; clients honour Retry-After. Expected |
| 9 | kubectl get hangs for exactly 60 seconds then errors | Hit --request-timeout; the API server accepted the request but storage (etcd) is slow or unhealthy |
| 10 | Watches in a custom client disconnect every ~30–60 minutes | --min-request-timeout randomised watch lifetime; reconnect from the last resourceVersion, which client-go does for you |
| 11 | A custom watcher with a narrow label selector loses its place after idle periods | It never advances its version because no matching events arrive; request allowWatchBookmarks=true |
| 12 | kubectl get pods -w prints every Pod again right after start | Those are the synthetic ADDED events of “get state and start”; use --watch-only to suppress |
| 13 | Request with resourceVersion=999999999 returns 504 “Too large resource version” | The version does not exist yet on this cluster; drop the parameter or use a version from a recent list |
| 14 | A controller “missed” a delete and still shows the object | Its cache is stale or it never relisted; a resync or restart will relist and converge. Level-triggered reconcile does not need the event |
| 15 | After upgrading from 1.33 to 1.34, API server memory during mass controller restarts drops sharply | WatchList (streaming lists) and streaming collection encoding are on by default from 1.34 |
Hands-On Lab: Watch the Machinery
Ten minutes on any cluster. Run each step and look for the fields named.
# 1. A list is a snapshot with a version
kubectl get --raw '/api/v1/namespaces/kube-system/pods?limit=2' | jq '.metadata'
# 2. Follow the continue token; the resourceVersion stays the same across pages
TOKEN=$(kubectl get --raw '/api/v1/namespaces/kube-system/pods?limit=2' | jq -r '.metadata.continue')
kubectl get --raw "/api/v1/namespaces/kube-system/pods?limit=2&continue=$TOKEN" | jq '.metadata'
# 3. Watch from that version and see typed events (Ctrl-C to stop)
RV=$(kubectl get --raw '/api/v1/namespaces/default/pods' | jq -r '.metadata.resourceVersion')
kubectl get --raw "/api/v1/namespaces/default/pods?watch=1&resourceVersion=$RV&allowWatchBookmarks=true&timeoutSeconds=20"
# 4. In another terminal, create and delete a Pod; observe ADDED, several MODIFIED, DELETED
kubectl run probe --image=busybox --restart=Never -- sleep 5
# 5. Ask for a version the server has already compacted (any small number)
kubectl get --raw '/api/v1/namespaces/default/pods?watch=1&resourceVersion=1&timeoutSeconds=5'
# {"type":"ERROR","object":{"kind":"Status","code":410,"reason":"Expired","message":"too old resource version: 1 (...)"}}
# 6. generation vs observedGeneration before and after a spec change
kubectl create deployment web --image=nginx
kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
kubectl set image deploy/web nginx=nginx:1.27
kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
# 7. Reproduce a 409
kubectl get deploy web -o json > a.json && kubectl scale deploy web --replicas=2 && kubectl replace -f a.json
# 8. See what kubectl really sends (Accept: Table, limit=500, no resourceVersion)
kubectl get pods -v=7 2>&1 | grep -E 'GET|Accept'Step 3 is the one that changes how you think about kubectl get -w: it is nothing more than this request, printed as a table.
Mistakes That Cost Points
- Treating
resourceVersionas a per-object counter. It is a cluster-wide storage revision. Two unrelated objects written back to back have adjacent values. - Confusing
resourceVersionwithgeneration. Status writes bump the first and not the second. “Has the controller seen my change” is ageneration/observedGenerationquestion. - Fixing a 409 with
--forceor by deleting the object. The fix is to re-read and reapply.replace --forcedeletes and recreates, changing theuidand orphaning anything that referenced it. - Retrying a 410 watch at the same version. The contract is relist, then watch from the new version.
- Expecting a watch to live forever. Watches are cut at a randomised point above
--min-request-timeout; clients reconnect. - Assuming
kubectl getoutput is a single atomic snapshot. With chunking it is a snapshot only while thecontinuetoken is valid; after about five minutes it may be inconsistent. - Thinking a filter (label or field selector) makes a watch cheaper to establish. Selectors are applied server-side, but the initial list still reads the whole collection from the cache; APF accounts for that cost.
- Reading
--event-ttlas watch history. It controlsEventobjects (the thingskubectl get eventsshows), not the five-minute etcd window.
Practise It Until the Errors Read Themselves
None of this is “advanced” in the sense of rare. Every cluster you touch is running thousands of these list/watch loops right now, and every CKA troubleshooting scenario that says “the Deployment is not updating” or “the controller logs show errors” is a question about one of the layers above. The skill the exam measures is recognising, from a single log line or a single error message, which layer is talking to you and what the correct next command is.
The KubeAstronaut mock exam bundle is built for that recognition: performance-based CKA and CKAD tasks that put you in front of a stuck rollout or a conflicting edit with a clock running, and KCNA/KCSA questions that check you can name the mechanism behind “controllers watch the API server”. Pair it with the object model guide for the vocabulary, the apply/create/replace guide for the write path, the kubectl patch guide for subresources, and the certification path guide to decide which of the five exams to sit next.
Frequently Asked Questions
What is a Kubernetes resourceVersion?
A string the API server attaches to every object and every list response that identifies the storage revision at which it was last written. For kube-apiserver it comes from the etcd revision, so it is a cluster-wide counter that increases with every write. Clients pass it back unmodified: as a precondition on updates (optimistic concurrency), as a consistency request on reads, and as the starting point of a watch.
What does “too old resource version” mean?
A watch (or an Exact list) asked for history the server no longer has. By default etcd keeps about five minutes of history, because the API server requests compaction every five minutes. The correct client response is to clear its cache, list again, and start a new watch from the resourceVersion that list returns. client-go’s reflector does this automatically.
What is the difference between resourceVersion and generation?
resourceVersion changes on every write, including status and metadata writes. generation changes only when the desired state (spec) changes, and controllers report the last generation they processed in status.observedGeneration. Tools such as kubectl rollout status compare the two to know whether a controller has seen your change.
Why does kubectl edit give a 409 Conflict but kubectl apply does not?
kubectl edit submits the object with the resourceVersion you opened, so any concurrent write invalidates it. Client-side kubectl apply sends a patch without that precondition and merges onto whatever is current. Server-side apply can still return a 409, but for a field-ownership conflict, which is a different mechanism.
What are watch bookmarks for?
A BOOKMARK event carries only a resourceVersion and tells the client that everything up to that version has been delivered. It lets a client whose watch sees few events keep its “last seen version” fresh, so that a reconnect lands inside the server’s history window instead of getting a 410. Request them with allowWatchBookmarks=true; the server does not guarantee how often, or whether, it sends them.
What are streaming lists and are they enabled?
A streaming list replaces the large initial list of list-then-watch with a watch that first sends a synthetic ADDED event per existing object, then a bookmark annotated k8s.io/initial-events-end: "true". On the API server the WatchList gate is beta and enabled by default since 1.34 (it was on in 1.32 and off in 1.33); the client-side WatchListClient gate is beta and on since 1.35.
How often does a watch disconnect?
At a randomised time above --min-request-timeout, which defaults to 1800 seconds, so between roughly 30 and 60 minutes. This is deliberate load spreading. Clients reconnect from their last seen resourceVersion.
Does kubectl get return a consistent snapshot?
A single response does. By default kubectl get paginates in chunks of 500 and all chunks share one resourceVersion, so the result is a consistent snapshot as long as every chunk is fetched before the continue token expires (about five minutes). After that the server offers an inconsistent continuation, which kubectl accepts. Use --chunk-size=0 when you need one unpaginated response.