Autoscale with HPA and Survive a Node Failure on Kubernetes

What you will build

A workload whose autoscaling is tuned from measurements rather than guesses, with evidence that it holds up under load and under failure.

Starting point: you only need a Vultr account. This challenge is self-contained, so you can do it without having done any other.

Part 1: The HorizontalPodAutoscaler

The HPA is the heart of this challenge. Do not accept defaults: choose every value and defend it with data.

• Metrics Server installed and working, with kubectl top returning real numbers
• An HPA on CPU utilisation, with resource requests set so the percentage means something
• A second signal, either memory or a custom metric such as requests per second
• minReplicas and maxReplicas you can justify
• A behavior section: a fast scaleUp policy and a slower scaleDown with a stabilization window so it does not flap

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

Part 2: Make it resilient

• Sensible resource requests and limits, justified by measurement
• Replicas spread across nodes with topology spread constraints or anti-affinity
• A PodDisruptionBudget
• A load test heavy enough to need more capacity than one node can give, so the node pool must grow too

The drill

1. Record a baseline at low load and note the HPA's current and desired replicas
2. Ramp up traffic and capture the timeline of CPU, replicas and pending pods
3. Find out what happens when the HPA wants more pods than the nodes can hold, and how you added capacity
4. Drain and delete a node during sustained load
5. Count failed requests and the time to full capacity
6. Change one HPA setting at a time, run again, and show the effect
7. Stop the load and show a clean, non-flapping scale-down

kubectl get hpa -w
kubectl describe hpa app
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data

How you will be judged

• Before and after numbers, not just configuration
• HPA settings chosen from the load test, with a reason for each
• Scale-down tested too, not only scale-up
• An honest account of what still falls over

Destroy the cluster and any load balancers or volumes it created when you are done so you are not billed for them.