October 21, 2026·4 min read
Most Kubernetes training follows the same pattern. A course, some slides, a workshop day, and a shared staging cluster that everybody is afraid to touch because something real runs on it.
Three months later the team still pages the same person when a pod will not start, and that person is now the single point of failure for an entire platform.
The reason is simple enough. Nobody learns infrastructure by reading about it. They learn it by breaking something and putting it back, which is exactly what a shared cluster forbids.
It is not timidity. On a shared cluster, an experiment can take down a colleague's work, and the blast radius of a mistake is other people's time. So engineers do the safe thing: they read the documentation, nod, and never run the command.
A cluster each removes the problem. The objection is cost, and it is a real objection if clusters are permanent. It mostly disappears if they exist for an afternoon.
That is the whole trick: training clusters should be created for the exercise and destroyed at the end of it. Nobody is paying for twelve idle control planes for the rest of the quarter.
Each of these takes under thirty minutes and teaches something that does not survive a slide.
Delete a node while a workload is running. Watch what reschedules and what does not. This is where people learn what a replica actually guarantees, and why a pod with local state is a different animal.
Fill the disk. Let them see eviction happen, read the events, and find out that the thing that got evicted is not always the thing that caused the problem.
Roll out a bad image, then roll it back. The deploy is the easy half. The lesson is in reading the rollout status and knowing when to stop it rather than waiting hopefully.
Break the readiness probe. A pod that is running and never takes traffic is the most confusing state in Kubernetes for anybody new, and nobody forgets it once they have caused it themselves.
Lose the config. Delete a ConfigMap or rotate a Secret under a running workload, and find out what actually reloads and what has been quietly serving stale values since.
Five drills, one afternoon, and afterwards the words in the documentation mean something.
Pairs, not an audience. Two people per cluster, swapping who is typing. A room watching one person's screen is a lecture with extra steps.
No slides first. Give the drill, let them fail for ten minutes, then explain. The explanation lands differently when they have already seen the symptom.
Write the runbook as you go. Every drill ends with three lines: what we saw, what we ran, what we would check first next time. That document is worth more than the training, and it is the thing that is still useful in six months.
Run it again in a quarter. Not the same drills: the same shape with different faults. Skills that are not used decay, and a team that has done this twice behaves very differently during a real incident.
It is not an interview, and it should not feel like one. The platform engineering version of these scenarios, with scoring and comparison, belongs in hiring: how to interview a platform engineer with a real Kubernetes cluster covers that, and the two should stay separate. The moment a training drill is graded, people stop experimenting and start performing.
It is also not a substitute for production experience. It is the thing that makes the first production incident survivable, which is different and cheaper.
For teams who are earlier than this, Docker basics is the better starting point: a cluster is a difficult place to learn what a container is.
The whole approach depends on clusters being disposable and identical. Identical, so that the person helping can say "run this" and have it mean the same thing on every screen. Disposable, so that breaking one is the point rather than a problem.
EasyEnv provisions a cluster per person from one recipe, with the workload already deployed and the fault already in place if you want it pre-broken. They come up in minutes, they are gone at the end of the session, and nothing anybody does reaches a real environment. Because every cluster comes from the same recipe, a drill you write once runs for every new engineer who joins next year.
The limits, plainly. Someone has to write the first set of drills, and they are better if they mirror your own stack rather than a generic example. An afternoon of a senior engineer's time is the real cost, more than the infrastructure. And for a team running one small cluster with two services, two drills are plenty: this is sized for teams who will be on call.
Nobody learns a readiness probe from a slide. They learn it from a pod that is running and refusing traffic while they work out why.
Give every engineer a cluster that exists for the afternoon, run five drills that break something real, write the runbook as you go, and do it again next quarter. It costs an afternoon and it is the difference between a team that pages one person and a team that does not.
Pre-configured Linux boxes, VS Code in the browser, and shared workspaces your team can join in one click. No more "works on my machine".
More posts you might like
Every engineer you hire proves they can work in a real environment. Nobody ever asks the people already there. Here is how to run an internal skills check that produces a training plan from evidence, finds your bus factor, and does not feel like an exam.
Contractors bill from day one but often cannot commit until week two, thanks to laptop setup and access sprawl. Ephemeral environments collapse that gap to an afternoon.
Read more
New tool, new stack, new skill gap. The right answer is sometimes hire and sometimes train. Here is how to choose.
Read more
