Every CKA candidate can create a Service. Most can explain that a ClusterIP is “virtual” and that kube-proxy “makes it work.” Far fewer can say what kube-proxy actually writes into the kernel, why a Service with healthy endpoints can still drop packets on one node, or why a LoadBalancer Service suddenly started failing health checks after someone set externalTrafficPolicy: Local.
That gap matters on the exam. The Services & Networking domain is 20% of the CKA, and its objectives include understanding connectivity between Pods and using ClusterIP, NodePort and LoadBalancer Services and endpoints. The scored tasks are practical — fix the Service, make the app reachable, preserve the client IP — but you solve them faster when you know the mechanism underneath. The CKA networking deep dive covers Service types, DNS, Ingress and NetworkPolicy; this article goes one layer down to the dataplane: how a Service’s virtual IP is turned into real forwarding rules, and every knob that changes which backend a packet reaches.
The Four-Step Chain Behind Every Service
Before touching any flags, fix this sequence in your head. It is the same for ClusterIP, NodePort and LoadBalancer:
- You create a Service with a selector. The API server allocates a ClusterIP from the
--service-cluster-ip-rangeand, for NodePort and LoadBalancer types, a node port from--service-node-port-range(default30000-32767). - The EndpointSlice controller (in kube-controller-manager) watches Pods matching the selector and writes their IPs, ports and readiness into EndpointSlice objects labelled
kubernetes.io/service-name=<svc>. - kube-proxy on every node watches Services and EndpointSlices and programs the node’s packet-forwarding rules — iptables, IPVS or nftables on Linux — so a packet to
ClusterIP:portis rewritten toPodIP:targetPort. - The CNI plugin carries the rewritten packet to the Pod, wherever it lives.
Nothing in that chain “proxies” traffic through a userspace process. kube-proxy is a controller that writes kernel rules; the kernel does the forwarding. That is why a dead kube-proxy does not break existing Services on a node — the rules stay in place — but stops new Services and endpoint changes from being reflected. It is a classic exam trap worth remembering.
EndpointSlices: The Source of Truth for Backends
The CKA no longer treats the legacy Endpoints object as primary. The Endpoints API has been deprecated since Kubernetes v1.33; EndpointSlice (stable since v1.21) is what kube-proxy, CoreDNS and modern load balancers read.
kubectl get endpointslices -l kubernetes.io/service-name=web -o wide
kubectl get endpointslice -l kubernetes.io/service-name=web -o yamlA slice looks like this (trimmed):
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: web-7xk2p
labels:
kubernetes.io/service-name: web
endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
addressType: IPv4
ports:
- name: http
port: 8080
protocol: TCP
endpoints:
- addresses: ["10.244.1.17"]
nodeName: worker-1
zone: eu-west-1a
conditions:
ready: true
serving: true
terminating: falseFacts the exam can reasonably expect:
| Detail | Value | Why it matters |
|---|---|---|
| Default endpoints per slice | 100 (--max-endpoints-per-slice, max 1000) | Large Services span several slices; always filter by the service-name label |
| Legacy Endpoints cap | 1000 endpoints, then truncated with endpoints.kubernetes.io/over-capacity: truncated | Only affects consumers still reading Endpoints |
ready condition | Pod is Ready and not terminating | The default set kube-proxy forwards to |
serving condition | Pod is Ready, regardless of deletion | Used when all endpoints are terminating |
terminating condition | Pod has a deletion timestamp | Set the moment kubectl delete pod runs, before containers exit |
publishNotReadyAddresses: true | Marks every endpoint ready | Needed by StatefulSet peer discovery before Pods pass readiness |
The serving/terminating split is the mechanism behind graceful rolling updates. When a Pod is deleted it is removed from the ready set immediately, so new connections stop arriving while the container’s preStop hook and terminationGracePeriodSeconds play out — the sequence covered in the Pod lifecycle and graceful shutdown guide. If every endpoint for a Service is terminating, kube-proxy will fall back to the ones still serving rather than drop traffic.
Exam reflex: a Service that “has no endpoints” is almost always a selector/label mismatch or a targetPort that does not match the container port. Check kubectl get endpointslices -l kubernetes.io/service-name=<svc> before you touch kube-proxy.
kube-proxy Modes: iptables, IPVS and nftables
kube-proxy has one job but three Linux backends. The mode is set with --proxy-mode or, in kubeadm clusters, the mode: field of the KubeProxyConfiguration stored in the kube-proxy ConfigMap in kube-system.
# kubeadm clusters: inspect the effective config
kubectl -n kube-system get cm kube-proxy -o yaml | grep -A1 -E '^\s*mode:'
# ask a running kube-proxy which mode it chose (metrics port 10249)
kubectl -n kube-system exec ds/kube-proxy -- \
wget -qO- http://127.0.0.1:10249/proxyMode
# or read the first lines of its log
kubectl -n kube-system logs ds/kube-proxy | grep -i -E 'proxier|proxy mode'iptables mode (today’s default)
kube-proxy installs chains in the nat table: KUBE-SERVICES matches on ClusterIP and port, jumps to a per-Service KUBE-SVC-* chain, which uses the statistic match to pick one KUBE-SEP-* (endpoint) chain at random, which performs the DNAT. NodePorts are matched in KUBE-NODEPORTS; traffic that needs SNAT is tagged via KUBE-MARK-MASQ.
# on a node: follow one Service through the rules
sudo iptables-save -t nat | grep -E 'KUBE-SVC|KUBE-SEP' | grep webTwo tuning parameters show up in configuration questions:
iptables.minSyncPeriod(default1s): the minimum gap between rule re-syncs so that bursts of endpoint changes are batched. Since v1.28 kube-proxy only rewrites the chains that changed, so the old advice to raise this to many seconds is obsolete.iptables.syncPeriod(default30s): how often kube-proxy does a full reconcile to repair rules something else deleted. Setting it very large is no longer recommended.
The weakness of iptables mode is linear rule evaluation: with tens of thousands of Services, every packet walks a long chain and every update rewrites a large ruleset.
IPVS mode (deprecated)
IPVS (IP Virtual Server) is a kernel load balancer with hash-table lookups, so it scales better than iptables and offers real scheduling algorithms:
ipvs.scheduler | Behaviour |
|---|---|
rr | Round robin (default) |
wrr | Weighted round robin |
lc / wlc | Least connections / weighted least connections |
sh | Source hashing — a crude form of client stickiness |
lblc, lblcr, sed, nq | Locality and shortest-expected-delay variants |
sudo ipvsadm -Ln # one virtual server per Service IP:port, real servers = endpointsThe current documentation is blunt about its future: the ipvs mode is deprecated as of v1.35, will be disabled by default in v1.40 (re-enable with the KubeProxyIPVS feature gate) and removed in v1.43. The kernel IPVS API turned out to be a poor fit for the Service model, and nftables delivers the same performance goals with a cleaner implementation. If an exam question or interviewer asks “which mode for large clusters?”, the forward-looking answer is nftables, not IPVS.
nftables mode (the successor)
The nftables backend (NFTablesProxyMode) went alpha in v1.29, beta in v1.31 and GA in v1.33. It requires kernel 5.13 or newer, processes endpoint changes faster than iptables and uses nftables maps and verdict maps so lookup cost does not grow linearly with the number of Services. In v1.37 the default is still iptables, but the documentation states plainly that a future release will flip the default to nftables — and recommends every cluster set mode explicitly so an upgrade never changes the backend by surprise.
# KubeProxyConfiguration excerpt
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables
nftables:
minSyncPeriod: 1ssudo nft list table ip kube-proxy | head -40Behavioural differences you should know before migrating a cluster from iptables to nftables:
- NodePort reachability: iptables mode binds NodePorts on every local IP; nftables defaults to
--nodeport-addresses primary, so NodePorts answer only on the node’s primary IPv4/IPv6 address. Override withnodePortAddresses: ["0.0.0.0/0"]if you depend on the old behaviour. - Firewall rules: iptables mode added accept rules for NodePorts to coexist with aggressive host firewalls; nftables mode adds nothing, so you must open the NodePort range yourself.
- Conntrack workaround: iptables mode installs a workaround for a pre-6.1 kernel bug that reset long-lived TCP connections; nftables does not. Check the
iptables_ct_state_invalid_dropped_packets_totalmetric and, if needed, run with--conntrack-tcp-be-liberal. - Localhost NodePorts (
127.0.0.1:3xxxx) are off in nftables mode; aKubeProxyNFTablesLocalhostNodePortsgate (alpha in v1.37) restores them.
Mode comparison for revision
| iptables | IPVS | nftables | |
|---|---|---|---|
| Status (v1.37) | Default, stable | Deprecated since 1.35; removal 1.43 | Stable since 1.33; future default |
| Kernel requirement | Any modern kernel | ip_vs modules | 5.13+ |
| Backend selection | Random (statistic match) | Configurable scheduler | Random |
| Scaling | Linear rule walk | Hash lookup | Map lookup |
| Inspect | iptables-save -t nat | ipvsadm -Ln | nft list table ip kube-proxy |
Some CNI plugins (Cilium is the common example) can replace kube-proxy entirely with eBPF programs. The CKA does not require configuring that, but if kubectl -n kube-system get ds kube-proxy returns nothing on a cluster, do not assume something is broken — check which CNI is installed, as described in the CNI awareness section of the networking deep dive.
Traffic Policies: Cluster vs Local
Two Service fields decide which nodes’ endpoints a packet may be forwarded to. They are independent and apply to different traffic sources.
externalTrafficPolicy (NodePort and LoadBalancer)
Cluster (default) | Local | |
|---|---|---|
| Eligible backends | All ready endpoints in the cluster | Only ready endpoints on the receiving node |
| Second network hop | Yes, when the backend is on another node | Never |
| Client source IP | Lost — kube-proxy SNATs to the node IP | Preserved |
| Load distribution | Even across Pods | Even across nodes, so uneven across Pods |
| Node with no local Pods | Still forwards (to another node) | Drops the traffic |
| LoadBalancer health check | NODE_IP:10256/healthz from kube-proxy | healthCheckNodePort, returns 200 only when a local endpoint exists |
Why the source IP is lost in Cluster mode: if node A receives a packet and forwards it to a Pod on node B, the reply must come back through node A for the connection to make sense. kube-proxy guarantees that by rewriting the source to node A’s IP. In Local mode the Pod is on the same node, so no rewrite is needed and the application sees the real client address — which is the whole reason externalTrafficPolicy: Local exists.
apiVersion: v1
kind: Service
metadata:
name: web
spec:
type: LoadBalancer
selector:
app: web
externalTrafficPolicy: Local
ports:
- port: 80
targetPort: 8080kubectl get svc web -o jsonpath='{.spec.healthCheckNodePort}{"\n"}'For a LoadBalancer Service with Local policy, Kubernetes allocates a healthCheckNodePort from the NodePort range. The cloud load balancer probes it on every node; kube-proxy answers 200 only on nodes that currently host a ready endpoint, so the load balancer stops sending traffic to nodes where it would be dropped. In Cluster mode the load balancer instead probes kube-proxy’s own health endpoint on port 10256, which returns 503 while a node is being deleted so connections can drain.
Since v1.28 (ProxyTerminatingEndpoints, GA), Local policy has one extra rule: if a node’s only local endpoints are all terminating, kube-proxy keeps forwarding to them instead of dropping. That bridges the gap during a rolling update between the last local Pod being deleted and the load balancer noticing the failed health check.
Exam scenario: “The application logs show every request coming from a 10.x node address. Make the real client IP visible.” Answer: set externalTrafficPolicy: Local on the NodePort/LoadBalancer Service. Then be ready to explain the trade-off — nodes without a replica stop serving, so pair it with a DaemonSet or with Pod topology spread constraints to keep a replica on each node.
internalTrafficPolicy (ClusterIP traffic)
Stable since v1.26, internalTrafficPolicy: Local tells kube-proxy to route in-cluster traffic to a Service only to endpoints on the same node as the client Pod. If the node has no local endpoint, the traffic is dropped — there is no fallback.
spec:
internalTrafficPolicy: LocalUse it for node-local agents exposed as a Service — a per-node log shipper, metrics sidecar or DNS cache deployed as a DaemonSet — where you want “talk to the copy on my node” semantics with a stable name. Do not use it for ordinary Deployments: a client on a node without a replica simply gets connection failures.
trafficDistribution: Preferring Nearby Endpoints Without Hard Failures
Traffic policies are binary — all endpoints or only local ones. spec.trafficDistribution (ServiceTrafficDistribution, GA in v1.33) adds a preference with graceful fallback, implemented through hints that the EndpointSlice controller writes and kube-proxy honours.
| Value | Behaviour | Status |
|---|---|---|
PreferSameZone | Use endpoints in the client’s zone; if the zone has none, fall back cluster-wide | GA (v1.35, via PreferSameTrafficDistribution) |
PreferSameNode | Use endpoints on the client’s node; fall back to same zone, then cluster-wide | GA (v1.35) |
PreferClose | Older alias for PreferSameZone | Deprecated |
apiVersion: v1
kind: Service
metadata:
name: api
spec:
selector:
app: api
trafficDistribution: PreferSameZone
ports:
- port: 80
targetPort: 8080Zone comes from the topology.kubernetes.io/zone label on the node that runs each endpoint Pod; the controller copies it into endpoints[].zone and writes hints.forZones on the slice.
How this differs from the older Topology Aware Routing annotation (service.kubernetes.io/topology-mode: Auto, GA in v1.33):
Automode tries to allocate endpoints to zones proportionally to allocatable CPU and applies safeguards — it needs at least roughly three endpoints per zone, every node must have a zone label, and it silently falls back to cluster-wide routing when balance is impossible. Predictable it is not.PreferSameZoneis deliberately simple: if a zone has endpoints, they get all that zone’s traffic. You own the risk of overloading a small zone.- If both are set, the annotation wins; the annotation is expected to be deprecated in favour of the field.
- A
Localtraffic policy takes precedence overtrafficDistributionfor that traffic direction; the field only guides routing when the corresponding policy isCluster.
For the exam, the decision table is short: need the client IP → externalTrafficPolicy: Local; need node-local only, failures acceptable → internalTrafficPolicy: Local; want lower cross-zone latency and egress cost but never a hard failure → trafficDistribution: PreferSameZone.
Session Affinity: Sticky Clients
By default each new connection to a Service is load-balanced independently. Setting sessionAffinity: ClientIP makes kube-proxy pin all connections from one client IP to the same endpoint for a window:
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800 # default: 3 hours; max 86400Two cautions that turn into trick questions. First, affinity is keyed on the source IP kube-proxy sees — with externalTrafficPolicy: Cluster on a NodePort, that may be a node IP after SNAT, so every external client collapses into one “session”. Second, affinity is per connection tuple in the kernel, so a Pod restart breaks stickiness anyway. Applications that need real sessions should externalise state; ClientIP affinity is a stopgap, not a design.
Masquerading, Conntrack and the Limits That Bite
A few kernel-side details explain otherwise mysterious failures:
- SNAT/masquerade. kube-proxy masquerades traffic that leaves the Pod network (NodePort
Clustermode, hairpin traffic from a Pod to its own Service) usingKUBE-MARK-MASQ.--masquerade-allforces SNAT for all Service traffic and is sometimes required by CNIs that cannot route Pod IPs symmetrically. - Hairpin traffic. A Pod connecting to a Service that load-balances back to itself needs the kubelet’s
hairpin-mode(hairpin-vethorpromiscuous-bridge) when using a bridge CNI. The official debug-service guide lists this as the cause when a Pod can reach every replica except itself. - Conntrack sizing. kube-proxy sets the connection-tracking table from
--conntrack-max-per-core(default32768) with a floor of--conntrack-min(131072), and TCP established entries expire after--conntrack-tcp-timeout-established(default24h). “Table full, dropping packet” indmesgon a busy ingress node means these are too small. - UDP and DNS. Conntrack entries for UDP outlive Pod churn; a stale entry can keep pointing DNS traffic at a dead CoreDNS Pod. The symptoms and the
ndotsinteraction are covered in the CoreDNS troubleshooting guide.
A Debugging Sequence for “Service Has Endpoints but Traffic Fails”
The official debug-service checklist is long; this is the condensed order that works under exam time pressure.
# 1. Does the Service resolve and have the ports you think?
kubectl -n app get svc web -o wide
kubectl -n app run tmp --rm -it --image=busybox:1.36 --restart=Never -- \
nslookup web.app.svc.cluster.local
# 2. Does it have READY endpoints? (selector, targetPort, readiness)
kubectl -n app get endpointslices -l kubernetes.io/service-name=web -o wide
kubectl -n app get pods -l app=web -o wide --show-labels
# 3. Do the Pods answer directly? (bypasses kube-proxy)
kubectl -n app exec deploy/client -- wget -qO- --timeout=2 http://10.244.1.17:8080/
# 4. Is kube-proxy healthy on the node where the client runs?
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide
kubectl -n kube-system logs <kube-proxy-pod> --tail=50
curl -s http://<node-ip>:10256/healthz # 200 = programming rules fine
# 5. Are the rules actually there?
sudo iptables-save -t nat | grep -c KUBE-SVC # iptables mode
sudo nft list table ip kube-proxy | grep -c 'service' # nftables mode
# 6. Is a traffic policy excluding this node?
kubectl -n app get svc web -o jsonpath='{.spec.internalTrafficPolicy} {.spec.externalTrafficPolicy}{"\n"}'
# 7. Is a NetworkPolicy, not the Service, blocking it?
kubectl -n app get networkpolicyStep 6 is the one people skip. A Service with internalTrafficPolicy: Local works perfectly from Pods on nodes that host a replica and times out from everywhere else — which looks exactly like a flaky network until you read the spec. Step 7 hands off to the NetworkPolicy guide; a default-deny policy in the namespace produces identical symptoms.
Common Mistakes on This Topic
- Restarting kube-proxy to “fix” a Service with zero endpoints. kube-proxy only programs what EndpointSlices contain; fix the selector or readiness probe.
- Setting
externalTrafficPolicy: Localon a Deployment with two replicas across ten nodes. Eight nodes now drop traffic. CheckhealthCheckNodePortbehaviour and spread replicas. - Expecting
internalTrafficPolicy: Localto fall back. It does not. UsetrafficDistribution: PreferSameNodewhen you want preference with fallback. - Assuming
sessionAffinity: ClientIPsees real client IPs behind a NodePort. Not withexternalTrafficPolicy: Cluster. - Choosing IPVS for a new cluster “because it scales.” It is deprecated; nftables is the supported path, with explicit
mode:set in the config. - Reading
Endpointsinstead ofEndpointSlices. The old object truncates at 1000 and is deprecated since v1.33. - Forgetting that nftables mode changes NodePort binding to the primary node IP only.
- Debugging on the wrong node. kube-proxy state is per node; test from a Pod on the node that is failing.
How to Practise This
Build a two-node kubeadm cluster (the lab setup guide walks through it) and run these experiments — each takes under ten minutes and maps to a likely task:
- Deploy a 2-replica app with a NodePort Service, hit the NodePort on a node without a replica, then set
externalTrafficPolicy: Localand watch it stop answering. ReadhealthCheckNodePort. - Switch kube-proxy to
nftablesby editing thekube-proxyConfigMap and restarting the DaemonSet; confirm with the/proxyModeendpoint andnft list table ip kube-proxy. Note what happened to NodePorts on secondary IPs. - Label your nodes with
topology.kubernetes.io/zone, settrafficDistribution: PreferSameZone, and inspecthints.forZoneson the EndpointSlice. - Delete a backend Pod while running a request loop; watch
terminating: trueappear on the slice before the Pod disappears.
Once the mechanics feel automatic, the remaining risk is speed under exam conditions. Sailor.sh’s Certified Kubernetes Administrator (CKA) Mock Exam Bundle puts Service and kube-proxy troubleshooting tasks inside a real multi-node cluster with the same remote-desktop constraints as the live exam, so you find out whether your debugging sequence holds up at minute 95 — and the free CKA practice questions are a good place to start before committing to a full timed run.
Frequently Asked Questions
What does kube-proxy actually do?
It watches Services and EndpointSlices and programs kernel packet-forwarding rules (iptables, IPVS or nftables on Linux) so that traffic to a Service’s virtual IP is rewritten to a backend Pod IP. It does not forward packets itself; the kernel does.
Which kube-proxy mode should I use in 2026?
Set the mode explicitly. iptables is still the default in v1.37 and works everywhere; nftables has been GA since v1.33, scales better and is slated to become the default. IPVS is deprecated since v1.35 and scheduled for removal in v1.43, so avoid it for new clusters.
What is the difference between externalTrafficPolicy Local and Cluster?
Cluster sends external traffic to any ready endpoint, may add a second hop and replaces the client source IP with the node IP. Local only uses endpoints on the receiving node, preserves the client IP, and drops traffic on nodes with no local endpoint; LoadBalancer Services get a healthCheckNodePort so the cloud load balancer avoids those nodes.
When should I set internalTrafficPolicy to Local?
When clients should talk only to the replica on their own node — typically a DaemonSet-backed agent — and a hard failure is acceptable if no local replica exists. For a soft preference, use trafficDistribution: PreferSameNode instead.
What is trafficDistribution PreferSameZone and how is it different from topology-aware hints?
PreferSameZone routes to endpoints in the client’s zone when any exist and otherwise falls back cluster-wide, with no safeguards. The topology-mode: Auto annotation allocates endpoints proportionally to zone CPU and refuses to apply hints when balance is impossible. PreferClose is a deprecated alias for PreferSameZone; PreferSameNode adds a node-first preference. All are GA as of v1.35.
Why does my Service show endpoints in EndpointSlices but kubectl get endpoints shows fewer?
The legacy Endpoints object is capped at 1000 addresses and annotated endpoints.kubernetes.io/over-capacity: truncated. EndpointSlices have no such cap (100 per slice by default, across as many slices as needed). The Endpoints API is deprecated since v1.33.
How long does sessionAffinity ClientIP stick?
By default 10800 seconds (3 hours), configurable up to 86400 via sessionAffinityConfig.clientIP.timeoutSeconds. Stickiness is keyed on the source IP kube-proxy observes, which may be a node IP after SNAT.
Is this level of detail really on the CKA?
The Services & Networking domain (20%) requires using ClusterIP, NodePort and LoadBalancer Services and endpoints, and the Troubleshooting domain (30%) covers networking failures. You will not be asked to recite flag defaults, but you will be asked to make a Service work or preserve a client IP, and knowing the mechanism is what makes those tasks fast.
Conclusion & Next Steps
A Service is three objects and one controller working together: the Service that allocates the virtual IP, the EndpointSlices that list ready backends, and kube-proxy translating both into kernel rules. Every knob in this article narrows the set of backends a packet can reach — traffic policies by node, trafficDistribution by topology preference, sessionAffinity by client — and every one of them has a failure mode that looks like “the network is broken” until you read the Service spec.
Commit the four-step chain, the Cluster/Local table and the mode comparison to memory, then go break a cluster on purpose. When you can predict which node will drop traffic before you run curl, the Services & Networking domain stops being the part of the CKA you hope they do not ask about. Pair this article with the exam domains breakdown to see where networking tasks sit in your overall time budget, and use timed mock exams to confirm the debugging sequence survives the clock.