What actually happens after a packet enters a Kubernetes node?

I used to fix Kubernetes networking issues the usual way: read the docs, change some YAML, restart the Pod. The problem would go away. But I hadn't learned anything — I couldn't tell you why it went away.

One evening I decided to stop asking "which YAML should I change?" and start asking a better question: where is this packet right now?

This is what I found.

The trace

Say a request arrives for a Service at 10.96.0.20:80. That IP is a lie — a useful one. No interface anywhere owns it. Here is the packet's actual path:

1. It arrives at eth0

The node's real NIC receives the frame. You can watch this happen:

tcpdump -i eth0 host 10.96.0.20

At this moment the destination is still the Service IP. Something has to translate it, because no Pod is listening on 10.96.0.20.

2. iptables rewrites reality

Before routing, the packet hits the PREROUTING chain in the nat table. Kubernetes hooks in here:

iptables -t nat -L KUBE-SERVICES | grep 10.96.0.20

The KUBE-SERVICES chain matches the Service IP and jumps to a KUBE-SVC-* chain, which picks one backend (with plain probability rules), and jumps again to a KUBE-SEP-* chain that performs DNAT: destination 10.96.0.20:80 becomes a real Pod IP, say 10.244.1.7:8080.

Two things surprised me here:

• kube-proxy never touches the packet. I always imagined it as a proxy in the data path. It isn't. It's a rule writer — it watches the API server and keeps these iptables chains in sync. The kernel does the actual forwarding.
• conntrack remembers the decision. The first packet of a connection goes through rule evaluation; the kernel records the translation in the connection tracking table (conntrack -L). Every following packet in that flow — and the replies coming back — get translated by the conntrack entry, not by re-running the rules.

3. Routing sends it to the bridge

Now the destination is a Pod IP. The node's routing table (ip route) says the Pod subnet lives behind the CNI bridge — cni0 here (your CNI may name it differently, and some CNIs don't use a bridge at all).

4. Across the veth pair, into another universe

Each Pod has its own network namespace — its own interfaces, routes, and iptables, fully isolated. The bridge connects to it through a veth pair: two virtual interfaces joined like a cable. One end (vethX...) is plugged into the bridge in the node's namespace; the other end appears inside the Pod's namespace with the boring name eth0.

You can stand on both sides of this boundary:

ip link show type veth            # node side
nsenter -t <pod-pid> -n ip addr   # inside the pod's namespace

The first time I ran ip addr inside a Pod's namespace and saw a completely different network stack, Kubernetes networking stopped being magic. It's namespaces and cables.

5. The socket

Inside the Pod's namespace, the packet reaches eth0 with destination 10.244.1.7:8080, where the application is actually listening. The reply walks the same path backwards, and conntrack un-DNATs the source so the client sees the reply come from the Service IP it originally called.

What I understood

• A Service IP is a rendezvous point that exists only in iptables rules. Nothing listens on it.
• kube-proxy is a control-plane component that happens to run on every node. The data plane is the Linux kernel.
• Pod isolation is network namespaces; Pod connectivity is veth pairs and a bridge; Service abstraction is NAT + conntrack. Three old, boring, well-documented Linux primitives — composed.
• Debugging changed permanently for me. Now every networking issue becomes the same question, asked at each hop: where is the packet right now, and what does the kernel believe about it?