Right-size Kubernetes: the playbook that took a cluster from 7 to 4 nodes

Most clusters run on guesses: requests copied from a tutorial, node pools sized for a launch two years ago, and a cloud bill nobody can explain line by line. Every month you pay for capacity that sits idle, and every month it feels too risky to touch. Right-sizing fixes that without changing anything your customers see. This is the exact order I use, with the queries, manifests and commands, and it's how one production cluster went from 7 to 4 nodes with no customer noticing.

Dominik Heinrich

November 6, 2026Updated November 7, 202611 min read

1. Get a baseline you can trust

Before changing anything, know what each node pool costs and who it belongs to. The FinOps Foundation calls this the Inform phase, and the cost optimization pillar of the Azure Well-Architected Framework starts in the same place: you can't optimize what you can't attribute. In Azure, Cost Management with amortized cost shows the real price per node pool, reservations included. Label every namespace with team and product, so a shared cluster can be split fairly and every euro has an owner.

Then compare three numbers per namespace: what is requested, what is actually used and what the nodes provide. Requests are what the scheduler reserves and what you pay for; usage is what the containers really consume. In many clusters requests sit far above real usage, and that gap is exactly what you pay for. Write the three numbers into a simple table per namespace. It becomes your before picture, and later your proof that the change worked.

Tip from practice

Use amortized cost, not actual cost. Otherwise a reservation bought in January makes January look expensive and the rest of the year look cheap.

Architecture of the AKS right-sizing loop: Azure Monitor managed Prometheus measures p95 usage, FluxCD applies new requests from Git, the cluster autoscaler shrinks the node pool, Cost Management sends a daily report, and reservations are bought for the smaller footprint. — The right-sizing loop on AKS: measure p95, change through Git, let the nodes follow, report daily, commit last.

2. Measure p95 over real weeks

Pull CPU and memory per container from Prometheus over two to four weeks, and make sure that window includes a month-end close, a campaign or whatever your busiest day is. On AKS, the Azure Monitor managed service for Prometheus collects these metrics without you running Prometheus yourself. Size on the 95th percentile, not the average: the average hides exactly the peaks that cause throttling and out-of-memory kills.

Treat CPU and memory differently, because Kubernetes does. CPU is compressible: a container that wants more than its share just runs slower. Memory isn't: a container above its memory limit gets killed, and the pod restarts. So for memory, also look at the maximum working set, not only the p95. Run the queries per container, export the result to a sheet and sort by the gap between request and p95. The biggest gaps at the top are where you start.

promql

# p95 CPU (cores) per container over the last 14 days
quantile_over_time(0.95,
  sum by (namespace, pod, container) (
    rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m])
  )[14d:5m]
)

# Peak working-set memory (bytes) per container over the last 14 days
max_over_time(
  container_memory_working_set_bytes{container!="", container!="POD"}[14d]
)

Tip from practice

Don't sum memory across pods over a long range. During rollouts old and new pods overlap, and the sum shows usage that never existed. Take the maximum per container instead.

3. Set requests and limits from the data

Set CPU requests at roughly p95 plus 10 to 20 percent headroom, and memory requests at the observed maximum plus a similar buffer. Keep memory limits equal or close to the request, so one leaky pod can't starve its neighbours on the same node. For CPU, many teams drop the limit entirely and rely on requests for fair sharing, which avoids throttling latency-sensitive services; keep a CPU limit where you need hard isolation, for example for batch jobs next to an API.

Ship the change like any other code change. Put the new values into the Helm values or Kustomize overlays in Git and let your GitOps tool, FluxCD or Argo CD, roll them out. Go one namespace at a time, starting with internal tools and staging, and watch latency, error rates and restarts for a day before you move on. If something looks off, the way back is a revert of one commit. That calm, reversible rhythm is what makes people trust the process.

yaml

# Helm values for one service, derived from 14-day p95 and max
resources:
  requests:
    cpu: 250m        # p95 was ~210m
    memory: 512Mi    # max working set was ~430Mi
  limits:
    memory: 512Mi    # equal to the request: no noisy neighbours
    # no CPU limit: avoids throttling, requests keep sharing fair

Tip from practice

The Vertical Pod Autoscaler in recommendation mode (updateMode: Off) gives you a second opinion per workload without changing a single pod.

Interactive

How many nodes do your requests really need?

Move the sliders to your numbers. Requests decide where pods fit, so oversized requests buy nodes nobody uses.

Pods60

CPU request per pod today500m

Real p95 CPU per pod150m

Headroom on top of p95+30%

Node size

nodes today

nodes after right-sizing (200m)

60%

less node capacity

CPU only, about 90% of a node is allocatable, at least 2 nodes for availability. Check memory the same way before you remove a node.

All calculators and checklists

4. Pick node pools that fit your pods

With honest requests you can finally see the real shape of your workload, and that shape decides the node sizes. Too small, and the per-node overhead of system pods, DaemonSets and the kubelet reservation eats a big share of every machine. Too big, and one half-empty node costs a lot while the autoscaler can't remove it. A useful check: how many of your typical pods fit on one node, and how much is left over? The AKS cost best practices recommend the same exercise before picking VM sizes.

Separate the work by how it behaves. Keep a small system node pool for cluster components, a user pool for production, and a spot node pool with a taint for interruptible work like batch jobs, CI runners and dev environments. Spot capacity can be taken back at short notice, so only workloads that tolerate it should ever land there. On AKS, node auto-provisioning, which is based on Karpenter, can also choose VM sizes for you based on what is pending.

5. Let the autoscaler do the shrinking

Once requests are honest and the pools fit, the scheduler packs pods tighter and some nodes end up mostly empty. The cluster autoscaler notices that and removes them, as long as the pods on them can move elsewhere. That's the moment the bill actually drops, so make sure the autoscaler is allowed to act: a sensible minimum and maximum per pool, and nothing that pins pods to specific nodes without a reason.

Watch for the classic blockers. Pods with local storage, pods without a controller, and strict PodDisruptionBudgets can all stop a node from being removed. The autoscaler logs tell you which pod blocked which node, and fixing those few cases is often worth more than another round of tuning requests. Give it a few days and you'll see the node count settle at a new, lower level.

Tip from practice

If the node count doesn't move after a week, read the autoscaler's scale-down events before touching anything else. It almost always names the one pod that is in the way.

Workload Autoscaling, AKS Cost Management Series, Episode 3, Azure Kubernetes Service (AKS) (YouTube). Loads from YouTube only after you click.

6. Remove nodes safely

Sometimes you want to speed things up and take nodes out yourself, for example to move to a new VM size. Before nodes go, check that every important workload has at least two replicas, a PodDisruptionBudget and sensible anti-affinity, otherwise a drain can take a service down. Then remove capacity one node at a time: cordon, drain, watch error rates and latency, wait, repeat.

Write the old minimum node count down as your way back, and lower the autoscaler's floor only after the cluster has run calmly on fewer nodes for a while. In one production cluster this took us from 7 to 4 nodes, and no customer noticed. The work itself was unspectacular, which is exactly the point: small steps, clear signals, and a revert at hand.

bash

# Which workloads would block or break a drain?
kubectl get pdb -A

# One node at a time; check dashboards before the next one
kubectl cordon aks-userpool-12345678-vmss000003
kubectl drain aks-userpool-12345678-vmss000003 \
  --ignore-daemonsets --delete-emptydir-data --timeout=10m

# After a calm week on fewer nodes, lower the autoscaler floor
az aks nodepool update -g rg-platform --cluster-name aks-prod -n userpool \
  --update-cluster-autoscaler --min-count 2 --max-count 6

Tip from practice

Drain during working hours with people watching the dashboards, not at 3 a.m. You want eyes on the graphs when something unexpected moves.

7. Commit only to the smaller footprint

Only now buy reservations or savings plans, sized to the cluster you ended up with, not the one you started with. Both the Azure and the AWS Well-Architected cost pillars make the same point in different words: optimize what you use before you optimize the price you pay for it. Cover the stable base with one- or three-year commitments and leave the peaks on pay-as-you-go or spot.

If you already hold reservations for VM sizes you no longer run, exchange or return them right away where your provider allows it, instead of letting them run idle. Commit last, not first: in the engagement behind this playbook, reservations were the final step after right-sizing and clean-up, and together the Azure run-rate fell 48% in five weeks.

8. Keep it small: make it a routine

Clusters grow back quietly: a new service copies generous requests from an old one, a load test leaves a node pool scaled up, a debug flag doubles the logs. The FinOps Foundation's Operate phase is about exactly this: turning a one-off clean-up into a habit. Send a short daily cost report to the team chat, set budgets with forecast alerts, and add a ResourceQuota per namespace so requests can't drift without anyone noticing.

Then put the p95 review in the calendar every quarter. Once the queries exist, it takes about an hour, and it usually finds one or two services that crept up since last time. The teams that keep their savings aren't the ones with the cleverest tooling; they are the ones that look at the numbers every day and treat cost like latency or errors, as one more signal of a healthy platform.

Takeaway
Right-sizing is a loop, not a project: measure p95, change one namespace, watch, let the nodes follow, then commit. Done in that order, the savings stay.

Real projects behind this

Try it yourself

Want this in your company? Let's talk mail@dominikheinrich.ch

Sources and official guides

Right-size Kubernetes: the playbook that took a cluster from 7 to 4 nodes

1. Get a baseline you can trust

2. Measure p95 over real weeks

3. Set requests and limits from the data

How many nodes do your requests really need?

4. Pick node pools that fit your pods

5. Let the autoscaler do the shrinking

6. Remove nodes safely

7. Commit only to the smaller footprint

8. Keep it small: make it a routine

Get in touch