Kubernetes in Production: 10 Things We Wish We Knew Earlier

Running a handful of Kubernetes clusters in a workshop is nothing like running twenty of them across client environments with real traffic, real on-call rotations, and real 2am pages. Here are the lessons that cost us the most time to learn — so you don't have to relearn them the hard way.

1. RBAC will bite you before networking does

Most teams over-invest in NetworkPolicy tuning early and under-invest in RBAC. In practice, the incidents that actually hurt come from over-permissioned service accounts — a CI pipeline with cluster-admin, a debugging pod left with a token that can read every Secret in the namespace. Scope roles per-namespace and per-workload from day one; retrofitting RBAC onto a live cluster is miserable.

2. Set resource requests before you ever touch autoscaling

The Horizontal Pod Autoscaler and cluster autoscaler are only as good as the requests/limits you feed them. We've seen HPA configs that looked perfect on paper thrash uselessly because requests were left at defaults. Get requests right first — based on actual observed usage, not guesses — then layer autoscaling on top.

3. Liveness probes without readiness probes cause cascading restarts

A liveness probe that fires during a slow dependency call will kill and restart a perfectly healthy pod, often making an already-degraded service worse. Always pair liveness with a readiness probe, and make liveness checks dumb and cheap — they should only catch true deadlocks, not slowness.

4. PodDisruptionBudgets save you during node upgrades, not just deploys

Teams usually add PDBs to protect against rolling deploys and forget they're just as critical during node drains for cluster upgrades or spot-instance reclamation. No PDB means an upgrade can take your whole service down at once instead of gradually.

5. etcd performance is your ceiling, not compute

Clusters that feel sluggish are rarely CPU-starved — they're usually waiting on etcd. Watch etcd disk latency and object counts (especially Events and CRDs) before you reach for bigger node pools.

6. Namespace-per-environment beats cluster-per-environment until it doesn't

Separate clusters per environment cost more but buy real isolation — a noisy-neighbor incident or a bad CRD install in staging can't touch production. We move teams to cluster-per-environment the moment compliance or blast-radius requirements show up; before that, namespaces are fine.

7. HPA on custom metrics needs a metrics pipeline you actually trust

Scaling on CPU is easy. Scaling on queue depth or request latency requires a metrics pipeline (usually Prometheus + an adapter) that itself needs to be highly available — if your metrics source flaps, your autoscaler flaps with it.

8. Image pull policy and registry rate limits will surprise you at the worst time

`imagePullPolicy: Always` on every deploy across a large node pool can hit registry rate limits during a mass rollout or node replacement event. Pin digests where you can and understand your registry's throttling behavior before you scale past a few dozen nodes.

9. Observability has to include the control plane, not just workloads

It's easy to instrument application pods and forget the control plane itself — API server latency, scheduler queue depth, controller-manager errors. When something feels globally slow, that's usually where the answer is.

10. Your upgrade strategy is a design decision, not an afterthought

Decide up front whether you're doing in-place minor upgrades, blue-green cluster replacement, or something in between — and rehearse it in a non-production cluster. The teams that get burned are the ones treating the first real upgrade as the rehearsal.

None of this is exotic — it's just what breaks first when a cluster goes from a demo to a system other people depend on. If you're heading into that transition, our Kubernetes & Containerization service page has the checklist we actually run engagements against.