Cluster Maintenance

Cluster Maintenance Quick Notes

Cordon, drain, uncordon

Use these before an OS patch or a kubelet upgrade on a worker.

kubectl drain node01 --ignore-daemonsets --delete-emptydir-data
kubectl uncordon node01

• DaemonSet Pods are left in place. Drain stops and complains while any DaemonSet Pod is on the node, because the controller would recreate it immediately. --ignore-daemonsets tells drain to proceed and still does not delete those Pods. kube-proxy and the CNI agent keep running until the node actually reboots or you delete them yourself.
• emptyDir is deleted with the Pod. Drain refuses unless you pass --delete-emptydir-data.
• Pods with no controller (a naked Pod) are not recreated. Drain refuses them unless you pass --force.
• PodDisruptionBudget: minAvailable or maxUnavailable. Drain blocks while evicting would violate the budget. Fix the budget or scale the workload before you retry. kubectl describe pdb shows ALLOWED DISRUPTIONS.
• Control plane: Draining the only control-plane node takes the API down with it. For a single control plane, plan the window. For HA, drain one control-plane node at a time and wait until etcd and the apiserver are healthy again.

Unschedulable in kubectl describe node and the taint node.kubernetes.io/unschedulable:NoSchedule both appear after cordon.

Version skew

Kubernetes minor versions must stay close:

• kube-apiserver is the version you quote as “the cluster version.”
• kube-scheduler and kube-controller-manager may trail the apiserver by at most one minor. They must not be newer than the apiserver.
• kubelet may trail the apiserver by up to three minor versions (the n−3 policy). A kubelet must not be newer than the apiserver it registers with.
• kubectl should be within one minor of the apiserver, newer or older.
• Upgrade one minor at a time. kubeadm will not jump from 1.33 to 1.35 in a single apply. Step 1.33 → 1.34 → 1.35.
• Control plane first, then workers. Workers may lag. The control plane must not lag behind the workers.

kubectl version
kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion
kubeadm version

Upgrade order

1. Read kubectl get nodes and kubeadm upgrade plan on the control plane.
2. Upgrade the primary control plane (kubeadm upgrade apply <version>). On a stacked etcd this also moves the local etcd. Take an etcd snapshot first.
3. Upgrade additional control planes one at a time (kubeadm upgrade node after the first apply has published the new version).
4. For each worker: kubectl drain, upgrade kubeadm/kubelet/kubectl, kubeadm upgrade node, restart kubelet, kubectl uncordon.
5. Confirm every node reports the new kubelet version and Ready.

Debian-based nodes, first control plane:

apt-get update && apt-cache madison kubeadm
apt-get install -y kubeadm=<pkg-version>
kubeadm upgrade plan
kubeadm upgrade apply v1.35.0
apt-get install -y kubelet=<pkg-version> kubectl=<pkg-version>
systemctl daemon-reload && systemctl restart kubelet

<pkg-version> looks like 1.35.0-1.1. upgrade apply runs only on the first control plane. Every other node is drained, then:

apt-get install -y kubeadm=<pkg-version>
kubeadm upgrade node
apt-get install -y kubelet=<pkg-version> kubectl=<pkg-version>
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon <node>

OS upgrades on a worker

1. kubectl drain node01 --ignore-daemonsets --delete-emptydir-data
2. Patch and reboot the VM.
3. Confirm systemctl status kubelet and that the node returns Ready.
4. kubectl uncordon node01.

Workloads with a controller come back on this node or another, according to scheduling. emptyDir contents are gone. Persistent volumes reattach if the Pod is rescheduled onto a node that can reach the disk.

etcd’s role in maintenance

• All cluster objects (minus what is stored only on node disk, such as container images and emptyDir) are in etcd.
• Stacked etcd: A member runs as a static Pod on each control-plane node. Data directory is usually /var/lib/etcd. Manifest: /etc/kubernetes/manifests/etcd.yaml.
• Quorum: A majority of members must agree before a write commits.

An even count does not tolerate more failures than the odd count below it. Add or remove members one at a time with etcdctl member add and etcdctl member remove so a leader remains.

• Take a snapshot before any control-plane upgrade or certificate operation you cannot easily redo.

Snapshot with etcdctl

etcdctl on a kubeadm node must present the etcd client certificate. API version 3 is required.

• Endpoint https://127.0.0.1:2379 matches the local member. Confirm the advertise URL in the etcd manifest if the lab uses a different host IP.
• Certificates live in /etc/kubernetes/pki/etcd/. server.crt is accepted for this local admin operation on kubeadm clusters. Some clusters issue a dedicated healthcheck-client pair. Use the paths in the etcd manifest if they differ.
• Copy the snapshot off the node if you are about to rebuild the disk.

Health of the live member:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

Restore

snapshot restore does not overwrite the live data directory. It writes a new data directory. You then point etcd at that directory.

Single control plane (one etcd member, the usual kubeadm layout):

sudo ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore

Then edit /etc/kubernetes/manifests/etcd.yaml:

• Change the hostPath that mounts the data directory from /var/lib/etcd to /var/lib/etcd-restore (both the volume and any --data-dir flag).
• The kubelet notices the manifest change and restarts the etcd static Pod.
• Wait until kubectl get pods -n kube-system shows etcd and the apiserver Running. The apiserver crashes until etcd answers; that is expected for a minute.

If the manifest uses a volume named etcd-data with hostPath.path: /var/lib/etcd, that path string is the one to replace. Read the file before you edit it.

HA restore is a different procedure: restore the snapshot on every member with the same --initial-cluster and --initial-advertise-peer-urls, and with --initial-cluster-state=new, then start the members together. A single member is the procedure above.

After a restore

• Objects match the snapshot time. Anything created later is gone.
• Node heartbeats catch up. Pods that the controllers still want are recreated.
• If the apiserver certificates were rotated after the snapshot and you also restored an old PKI, clients whose kubeconfig trusts the old CA still work only if that CA is the one on disk. Snapshot /etc/kubernetes/pki when you snapshot etcd if certificates may have changed in between.

Official docs

• Node maintenance / drain
• Version skew policy
• Upgrading kubeadm clusters
• etcd snapshot with kubeadm