A step-by-step guide to upgrading a Kubernetes cluster simplified
Source: www.cajieh.com
Pre-requisite:
- Basic knowledge of Container technology and kubernetes are required
- Proficient in the use of command line tools i.e. Bash terminal
- Access to a Kubernetes cluster if you want to experiment with the example in this tutorial
Introduction
Upgrading a Kubernetes cluster requires a sequential update of the control plane components followed by the worker nodes. This tutorial will guide you on how to utilize kubeadm, a built-in Kubernetes CLI tool, for the process. For the sake of simplicity, we'll consider a Kubernetes cluster composed of a control plane node and a worker node, both of which need to be upgraded to the latest version. Once you've accessed the control plane of the Kubernetes cluster, the first step is to check the nodes in the cluster.
Upgrade the controlplane node to the target version
Step #1: View cluster nodes
$ k get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 14d v1.30.0
node01 Ready <none> 14d v1.30.0
In our scenario, our Kubernetes cluster consists of a control plane and a worker node.
Step #2: Drain the controlplane node
This step ensures that the control plane node is safely evicted of workloads before the upgrade.
$ kubectl drain controlplane --ignore-daemonsets
pod/local-path-provisioner-75655fcf79-xw9b6 evicted
node/controlplane drained
This action would evict all the workloads such as pods in the cluster and uncordon the node.
Note: Throughout this tutorial, always remember to replace the name of component and version such as “controlplane” and “v1.30.0” with the actual name of your control plane node and version respectively. Also, in most command executions, we'll be showing the tail of the output.
Step #3: Plan the upgrade
Before starting the controlplane upgrade process, we have to plan the upgrade by checking the kubeadm version and the cluster target version to see if kubeadm requires an upgrade before the cluster node upgrade. This can be done using the kubeadm upgrade plan command which displays information about the current and target versions of the cluster components.
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.30.0
[upgrade/versions] kubeadm version: v1.30.0 // Current version
[upgrade/versions] Target version: v1.30.3 // Target version
[upgrade/versions] Latest version in the v1.30 series: v1.30.3
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT NODE CURRENT TARGET
kubelet controlplane v1.30.0 v1.30.3
kubelet node01 v1.30.0 v1.30.3
Upgrade to the latest version in the v1.30 series:
COMPONENT NODE CURRENT TARGET
kube-apiserver controlplane v1.30.0 v1.30.3
kube-controller-manager controlplane v1.30.0 v1.30.3
kube-scheduler controlplane v1.30.0 v1.30.3
kube-proxy 1.30.0 v1.30.3
CoreDNS v1.11.1 v1.11.1
etcd controlplane 3.5.12-0 3.5.12-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.30.3 // Target version
Note: Before you can perform this upgrade, you have to update kubeadm to v1.30.3.
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
__________________________________________________________________
For this step the full output is shown since it contains useful information for the cluster upgrade. In the output, we can see that the target version is v1.30.3, and that the current version of kubeadm doesn't match the target version. Therefore, we must first upgrade kubeadm before upgrading the other cluster components.
Step #4: Upgrade the kubeadm to match the target version
sudo apt-get update && sudo apt-get install -y kubeadm=1.30.3-1.1
Confirm that the kubeadm has been upgraded to 1.30.3
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.3", GitCommit:"6fc0a69044f1ac4c13841ec4391224a2df241460", GitTreeState:"clean", BuildDate:"2024-07-16T23:53:15Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}
Step #5: Upgrade the controlplane node
Run kubeadm upgrade apply command specifying the target version of v1.30.3 to upgrade the controlplane node.
sudo kubeadm upgrade apply v1.30.3
upgrade/versions] Cluster version: v1.30.0
[upgrade/versions] kubeadm version: v1.30.3
[upgrade] Are you sure you want to proceed? [y/N]: y // Type “y” to continue the upgrade
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.30.3". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
Step #6: Upgrade the controlplane node
Additionally, we need to upgrade the versions of kubelet and kubectl to match the target versions. The kubelet is an agent that runs on each node and is a key component of a Kubernetes node while kubectl is a CLI tool for interacting with a Kubernetes cluster.
sudo apt-get update && sudo apt-get install -y kubelet=1.30.3-1.1 && kubectl=1.30.3-1.1
Hit:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease
Hit:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb InRelease
----
The default action is to keep your current version.
*** kubelet (Y/I/N/O/D/Z) [default=N] ? y Type “y” to continue the process
---
The default action is to keep your current version.
*** kubelet (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /etc/default/kubelet ...
Step #7: Restart the kubelet
Next, we have to restart the kubelet using the following command:
sudo systemctl daemon-reload && systemctl restart kubelet
Step #8: View the nodes
Next, we check the nodes again to view the status of the control plane node that we just upgraded.
$ k get no
NAME STATUS ROLES AGE VERSION
controlplane Ready,SchedulingDisabled control-plane 14d v1.30.3
node01 Ready <none> 14d v1.30.0
In the output, we see that the controlplane node status is Ready,SchedulingDisabled. The Ready condition indicates that the node is healthy and available to accept pods, but the SchedulingDisabled condition means that no new pods can be scheduled to the node. This is expected because we previously drained the control plane, which subsequently cordoned off the node.
Step #9: Uncordon the controlplane node
Let’s uncordon the controlplane node to allow new pods scheduling and view the nodes status again.
kubectl uncordon controlplane
node/controlplane already uncordoned
$ k get no
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 14d v1.30.3
node01 Ready <none> 14d v1.30.0
Great! The upgrade process of the control plane node from v1.30.0 to v1.30.3 has successfully completed. Next, we'll look at how to upgrade the worker node in the following section.
Upgrade the worker node to the latest target version
In our scenario, the worker node is named node01 and it's a Linux-based system, so our instructions will be based on that. However, if you're upgrading a Windows-based system, please refer to the Kubernetes official docs on Upgrading Windows nodes
Step #1: SSH into the worker node
$ ssh node01
Step #2: View kubeadm version
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.0", GitCommit:"7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a", GitTreeState:"clean", BuildDate:"2024-04-17T17:34:08Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}
Step #3: Upgrade the kubeadm
Run the same kubeadm upgrade command that was previously used to upgrade the control plane node. See it again below
sudo apt-get update && sudo apt-get install -y kubeadm=1.30.3-1.1
Reading state information... Done
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0 to remove and 178 not upgraded.
Need to get 10.4 MB of archives.
Step #4: View kubeadm version
Verify the version again
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.3", GitCommit:"6fc0a69044f1ac4c13841ec4391224a2df241460", GitTreeState:"clean", BuildDate:"2024-07-16T23:53:15Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}
Step #5: Upgrade the local kubelet configuration for the worker node
Next, invoke the kubeadm upgrade command to upgrade the local kubelet configuration for the worker node.
$ kubeadm upgrade node
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
Awesome, the upgrade for the kubelet configuration was successful, but as indicated in the output of the kubeadm upgrade node, we also need to upgrade the kubelet and kubecli packages
Step #6: Drain the worker node
Initiate maintenance on the node by setting it as non-schedulable and removing the workloads
kubectl drain node01 --ignore-daemonsets
Step #7: Upgrade the kubelet and kubectl
sudo apt-get update && sudo apt-get install -y kubelet=1.30.3-1.1 && kubectl=1.30.3-1.1
The default action is to keep your current version.
*** kubelet (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /etc/default/kubelet ...
node01 $ k get no
Step #8: Restart the kubelet
sudo systemctl daemon-reload && systemctl restart kubelet
Step #9: Uncordon the worker node
Next, uncordon the worker node to allow new pod scheduling, and then view the node status again
kubectl uncordon controlplane
node/controlplane already uncordoned
Step #10: Exit the worker node
Exit node01 to go back to the controlplane node. Then, view nodes again:
node01 $ exit
controlplane $
controlplane $ k get no
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 14d v1.30.3
node01 Ready <none> 14d v1.30.3
Awesome! The controlplane and worker node have been updated from version v1.30.0 to v1.30.3, and other dependencies such as kubeadm, kubelet, and kubectl were updated to the same version. This concludes the steps for upgrading a Kubernetes cluster, composed of a control plane and a worker node, from its current version to the target version
Please note that this is a simplified guide. Depending on your specific setup and requirements, the upgrade process may involve additional steps or considerations. Always refer to the official Kubernetes documentation for the most accurate and detailed instructions.
I hope this helps! Go to the contact page and let me know if you have any further questions.
Happy learning!