Skip to content
prakashdass

2026-07-20 · trace

When I run kubectl apply, what really happens between my terminal and a running Pod?

kubectl apply -f deploy.yaml, and a few seconds later there's a Pod running. For a long time I treated that gap as a single magic step. Then I tried to name every component the request actually touches, in order — and the magic turned into a sequence of small, boring, well-defined jobs.

The question I kept asking: at each hop, who is doing the work, and what are they reacting to?

The trace

kubectlapply -fAPI servervalidate + admitcontrollersDeploy → RS → Podschedulerassign a nodekubeletmake it realetcdcluster statePodcontainers runningread / writestartsevery component watches the API server and reacts — nothing is pushed
From kubectl apply to a running Pod — a chain of watchers, not a call stack.

1. kubectl just talks to the API server

kubectl isn't special. It authenticates and sends a plain HTTP request to the API server. That's the only component it talks to — and, it turns out, the only component anything talks to.

2. The API server validates and persists — then stops

The API server authenticates the request, runs authorization and admission checks, validates the object, and writes it to etcd. And then, crucially, it does nothing else. It does not schedule the Pod. It does not start a container. Its job is: be the front door to the cluster's state, and keep that state in etcd. etcd is the single source of truth; everything else is a cache of it.

3. Controllers turn one object into many

A Deployment doesn't run anything by itself. The Deployment controller is a loop watching the API server; it sees the new Deployment and creates a ReplicaSet. The ReplicaSet controller sees that and creates the Pod objects — which at this point are just records in etcd with no node assigned. Each controller does one small translation and writes the result back through the API server.

4. The scheduler only decides where

The scheduler watches for Pods with no node. For each one it filters the nodes that could run it and scores the rest, then does exactly one thing: writes the chosen node name onto the Pod object. It doesn't contact the node. It doesn't start anything. It just records a decision.

5. The kubelet makes it real

Every node runs a kubelet, watching the API server for Pods assigned to its node. When it sees one, it finally does the physical work: it calls the container runtime (through the CRI) to pull the images and start the containers, sets up networking, and reports status back. Only here — at the very end — does software actually run.

What I understood

  • Nothing pushes; everything watches. kubectl → API server is the only direct call. After that, each component is an independent loop watching shared state and reacting. That's why Kubernetes is resilient: any component can restart and just resume reconciling.
  • The API server + etcd are the whole truth. Every other component is stateless-ish and rebuildable, because the real state lives in one place.
  • Each step does almost nothing. Controllers translate, the scheduler decides, the kubelet executes. The power is in composing tiny reconcilers, not in any one clever component.
  • "Declarative" finally clicked. I don't tell Kubernetes to do things. I write down what should be true, and a chain of watchers works to make it true and keep it true.