Skip to content
DRDiwakar Rai

Notes12-Troubleshooting

Troubleshooting

A repeatable order for a workload that will not run, a control plane that will not answer, a node that will not go Ready, and traffic that will not flow.

Updated Oct 1, 2026

Troubleshooting Quick Notes




Start wide, then narrow

  1. Which cluster? kubectl config current-context. kubectl config use-context <name> if this shell talks to more than one.
  2. Which namespace? kubectl get pods -A once, then stay on -n for the namespace that owns the object.
  3. What is the object’s phase and events? kubectl describe puts the useful lines under Events and Conditions.
  4. Change one thing, then re-check. Editing three manifests before the next kubectl get hides which edit mattered.


bash
kubectl config current-context
kubectl get nodes
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail


Application failures


bash
kubectl describe pod <name> -n <ns>
kubectl logs <name> -n <ns> --previous -c <container>
kubectl get pod <name> -n <ns> -o yaml


Status or event

What it usually means

What to change

Pending / FailedScheduling

No node fits. The scheduler skipped nodes that lack free CPU or memory for the requests, that carry a NoSchedule taint the Pod does not tolerate, or that fail nodeSelector / required node affinity. A PVC with WaitForFirstConsumer or the wrong zone does the same

Lower requests, add a toleration, fix node labels, or bind a volume the node can attach

ImagePullBackOff / ErrImagePull

Name, tag, or registry auth

image, imagePullSecrets, or imagePullPolicy: Never if the image is already on the node

CrashLoopBackOff

Process exits. The current log may only show the new start

kubectl logs --previous. Then command/args, missing env, failing liveness probe

CreateContainerConfigError

Env key points at a missing ConfigMap or Secret, or the key is absent

Create the object or fix secretKeyRef / configMapKeyRef

RunContainerError / permission

runAsNonRoot with an image that is UID 0, or a read-only root and the app writes to /

securityContext, or an emptyDir on the path it writes

OOMKilled

Memory limit exceeded (lastState.terminated.reason). BestEffort Pods (no requests or limits) are killed first under node pressure. Guaranteed Pods (request equal to limit for CPU and memory on every container) are killed last

Raise the memory limit, or set request equal to limit if this Pod must be Guaranteed

Init:CrashLoopBackOff

An init container is failing. App containers have not started

kubectl logs <pod> -c <init-name>

A Deployment that never finishes a rollout is the same list applied to the new ReplicaSet’s Pods:


bash
kubectl rollout status deployment/<name>
kubectl get rs -n <ns>
kubectl describe deployment/<name>


Progressing=False / ProgressDeadlineExceeded means the new Pods were not Ready in time. Fix the Pod, then kubectl rollout restart only if the template itself must be poked again.


Control plane failures


If kubectl get nodes times out or returns connection refused, the apiserver is not serving. SSH to the control-plane node.


bash
crictl ps -a | head -20
ls /etc/kubernetes/manifests
journalctl -u kubelet --since "20 min ago" --no-pager


Static Pod names look like kube-apiserver-controlplane. If crictl ps shows the container restarting:


bash
crictl logs <apiserver-container-id> 2>&1 | tail -50


Log or event

Fix

YAML parse error, unknown flag

The manifest was edited badly. Compare with a backup or kubeadm defaults. The kubelet restarts the Pod when the file is valid again

failed to create listener / address in use

--bind-address or hostNetwork port clash

etcd connection refused

etcd static Pod is down, or --etcd-servers points at the wrong URL. Fix etcd first. The apiserver will come back once etcd answers

x509: certificate has expired or not valid for

kubeadm certs check-expiration, then kubeadm certs renew all. openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text lists Subject Alternative Names. The certificate must include every name clients use (advertise IP, load-balancer hostname, kubernetes, kubernetes.default, the kubernetes Service ClusterIP). Restart the apiserver static Pod after renew, and copy a fresh admin.conf to ~/.kube/config if kubectl starts failing TLS

Manifest present, no container

kubelet cannot read the file (permissions, staticPodPath changed) or containerd is down

Scheduler and controller-manager crashes leave the API up and the cluster stale: no new Pods bind, no ReplicaSets reconcile. Check those two static Pods the same way. A leader-election failure often means they cannot reach the apiserver with /etc/kubernetes/scheduler.conf or controller-manager.conf.


Quick health of a living apiserver:


kubectl get --raw='/readyz?verbose'

/readyz plus the static Pod list is the health check. A single failed check names the broken dependency (etcd, for example).

etcd-specific:


bash
sudo ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key


Error: unhealthy or a data-dir disk full: df -h /var/lib/etcd. Nospc in the etcd log means the member will not elect until the disk has space. Take a snapshot before you delete anything you might need:


bash
sudo ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key


Restore writes a new directory (etcdctl snapshot restore /opt/etcd-backup.db --data-dir=/var/lib/etcd-restore). Point the etcd manifest hostPath and --data-dir at it. Do not restore on top of the live data directory.


Worker node failures



bash
kubectl describe node <node>
systemctl status kubelet
journalctl -u kubelet -u containerd --since "30 min ago" --no-pager


Node state

Usual cause

NotReady, NetworkPluginNotReady

CNI not installed, or the CNI DaemonSet is crash-looping. /etc/cni/net.d

NotReady, kubelet stopped

systemctl restart kubelet. If it exits, the unit log names the reason: swap, bad config, expired client cert

NotReady after reboot

kubelet or containerd disabled: systemctl enable --now kubelet containerd

Ready, SchedulingDisabled

The node is cordoned. kubectl uncordon when maintenance is finished

MemoryPressure / DiskPressure

Free inode and disk under /var/lib/containerd and /var/log. Eviction will keep killing Pods until you do

Conditions show a taint not-ready:NoExecute

Consequence of the kubelet failing heartbeats, not a taint you should delete by hand. Fix the kubelet

Kubelet client certificate problems show up as Unable to register node and x509 errors against the apiserver. On kubeadm the kubelet uses /etc/kubernetes/kubelet.conf. Certificate rotation is normally automatic (/var/lib/kubelet/pki). A restored VM with an old kubelet.conf is a reason to rejoin (kubeadm reset on that node only, then kubeadm join) rather than to copy another node’s certificates. kubeadm reset wipes the local cluster install. Run it only on the broken worker.


Pods stuck Terminating after a node died: the volume or the finalizer is waiting on a kubelet that is gone. If the node is not coming back, kubectl delete pod <name> --force --grace-period=0 clears the object so the controller can replace it.


Service and network failures


Split the path. Each hop fails differently.

  1. Pod IP from a peer Pod (same namespace). If this fails, the CNI or a NetworkPolicy is the problem. kube-proxy is not on this path.
  2. Service ClusterIP from a peer Pod. If Pod IP works and ClusterIP does not, check Endpoints and kube-proxy.
  3. DNS name. If the ClusterIP works by number and the name does not, CoreDNS is the problem.
  4. NodePort from outside. If ClusterIP works inside and the node port does not, look at kube-proxy, the node firewall, and the port you were given.


bash
kubectl get svc <name> -n <ns> -o wide
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name>
kubectl get pods -n <ns> --show-labels
kubectl get networkpolicy -n <ns>
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=30


What you see

What to fix

Endpoints empty, Pods Running

Service selector does not match Pod labels. Or readiness is failing, so they are not members

Endpoints filled, connection timeout

NetworkPolicy on the client or the server. A policy with ingress: [] denies all inbound. Egress without port 53 breaks DNS only

nslookup fails, Pod IP works

CoreDNS Pods, the coredns ConfigMap, or the kube-dns Service endpoints

CoreDNS CrashLoop

The Corefile. kubectl logs prints the parse error. Restore cluster.local and the kubernetes plugin

Ingress has no ADDRESS

No controller, or the controller Service is not LoadBalancer/NodePort. The Ingress object is only a route

Ingress 404 from the controller

host, path, pathType, or the backend Service name. kubectl describe ingress shows the backend

Works on-node, fails across nodes

Routes or the CNI DaemonSet not running on one of the nodes. Compare ip route and the plugin Pods

A debug Pod in the same namespace is a fair client:


bash
kubectl run net --rm -it --image=busybox:1.36 --restart=Never -- sh
# wget -qO- --timeout=2 http://web.shop.svc.cluster.local
# nslookup kubernetes.default


Use an image the cluster can already pull. wget to the Service and nslookup of kubernetes.default separate “this app is broken” from “cluster DNS is broken.”


Checks before you move on

  • The object is in the namespace you meant to change.
  • kubectl get shows Running / Ready / Bound, which is a stronger signal than “the YAML was applied.”
  • For a Service, the EndpointSlice has addresses.
  • For RBAC, kubectl auth can-i <verb> <resource> --as <user> -n <ns> prints yes.
  • For a node fix, kubectl get nodes shows Ready. If you drained the node, kubectl uncordon it when maintenance is finished.


Official docs