Upgrading Kubernetes

Upgrading Kubernetes ensures your cluster remains secure, stable, and compatible with the latest features. This guide explains how to upgrade Kubernetes using kubeadm on Ubuntu 24.04 LTS to the latest supported versions such as 1.31. Follow the process carefully to minimize downtime on production workloads.

Prerequisites

Before starting the upgrade, verify your environment meets all requirements. You need an existing Kubernetes cluster managed by kubeadm, administrative access via kubectl, and root privileges on all nodes. All nodes must run Ubuntu 24.04 LTS with sufficient disk space and network connectivity to registry.k8s.io.

Check current versions

Confirm the exact versions of your control plane components and nodes. Run the following command on the control plane node:

kubectl version --client=true
kubeadm version
kubelet --version

Expected output shows the current API server version and confirms kubeadm and kubelet are installed. Note the minor version difference between components, as upgrades must follow sequential minor version steps.

Backup etcd and configuration

Always create a full backup of etcd before any upgrade. Use the following command to snapshot the etcd data:

sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db 
  --endpoints=https://127.0.0.1:2379 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key

Store the snapshot securely. Also back up the entire /etc/kubernetes directory and any custom manifests.

Step-by-step upgrade process

1. Prepare package repositories

Update the Kubernetes package repository to access version 1.31 packages. Create or update the repository file:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Refresh the package index:

sudo apt update

2. Unhold current packages

Release version pinning on kubeadm, kubelet, and kubectl:

sudo apt-mark unhold kubeadm kubelet kubectl

3. Upgrade kubeadm

Install the new kubeadm binary first. This component orchestrates the upgrade:

sudo apt install -y kubeadm=1.31.0-1.1

Re-hold the package to prevent accidental updates:

sudo apt-mark hold kubeadm

Verify the new version:

kubeadm version

4. Plan the upgrade

Review the upgrade plan to identify component changes and any required actions:

sudo kubeadm upgrade plan

The output lists the target version, component images from registry.k8s.io, and any deprecated APIs. Resolve warnings before proceeding.

5. Apply control plane upgrade

Upgrade the control plane components in a single operation:

sudo kubeadm upgrade apply v1.31.0

This command updates the API server, controller manager, scheduler, and etcd if necessary. Monitor the output for successful pod restarts and certificate renewals.

6. Upgrade kubelet and kubectl

Install matching versions of kubelet and kubectl:

sudo apt install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl restart kubelet

7. Upgrade worker nodes

Repeat the package upgrade steps on each worker node. Then run the node upgrade command from the control plane:

sudo kubeadm upgrade node

Drain and cordon each node before restarting kubelet to avoid workload disruption:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
sudo systemctl restart kubelet
kubectl uncordon <node-name>

Common Errors

• Version mismatch between kubeadm and kubelet causing the upgrade to fail.
• Insufficient disk space on the control plane node during image pulls.
• etcd backup not performed, leading to data loss risk.
• Network policies blocking access to registry.k8s.io during component image downloads.
• Forgetting to unhold packages before running apt install.

Going Further

• Automate upgrades using Ansible playbooks or Terraform with AWS Launch Templates for node replacement.
• Implement canary upgrades by adding new nodes at the target version and gradually migrating workloads.
• Integrate cluster upgrades into CI/CD pipelines with automated testing of critical applications.
• Monitor upgrade success with Prometheus metrics for API server latency and etcd health.

Conclusion

Following this structured approach allows you to upgrade Kubernetes reliably while maintaining cluster availability. Regular upgrades keep your environment secure and aligned with the latest Kubernetes features. Always test the process in a staging environment first.

FAQ

How often should I upgrade Kubernetes?

Plan minor version upgrades every 3 to 6 months. Stay within the supported skew between control plane and nodes to avoid compatibility issues.

Can I skip minor versions during upgrade?

No. Kubernetes requires sequential minor version upgrades. Skipping versions can cause etcd or API incompatibilities.

What happens if the upgrade fails midway?

Restore from the etcd snapshot created before the upgrade. Re-run kubeadm upgrade apply after fixing the root cause identified in the logs.

Do worker nodes require the same upgrade steps?

Yes. Worker nodes must run the same kubelet and kube-proxy versions as the control plane to maintain cluster stability.

Is zero-downtime upgrade possible?

With multiple control plane replicas and proper draining of nodes, you can achieve near-zero downtime for most workloads during the upgrade process.

Retour parcours Kubernetes — hub de la série et leçons sœurs.