Logging and Monitoring

Logging and Monitoring Quick Notes

Two different signals

• Logs: What the process wrote to stdout and stderr. They explain crashes, bad config, and application errors.
• Metrics: CPU and memory samples. They explain throttling, OOM, and “which Pod is hot.” kubectl top needs metrics-server. kubectl logs does not.

Container logs

The kubelet stores the latest stdout/stderr of each container and serves it through the API.

kubectl logs nginx
kubectl logs nginx -c sidecar          # pick a container in a multi-container Pod
kubectl logs nginx --previous          # last terminated container
kubectl logs nginx --tail=50 -f
kubectl logs deploy/web --all-containers=true
kubectl logs -l app=web --prefix --tail=20
kubectl logs nginx --since=10m

• --previous: The file from the container that just exited. This is the first command for CrashLoopBackOff. The current container’s log is often only the new start.
• --all-containers: Every container in the Pod, including sidecars. Init containers need -c <init-name> (they are not in the app container list).
• Deployment / Job / label: kubectl logs accepts a controller or a label selector and fans out.
• What you will not find here: Files the app wrote to its own disk. Those need kubectl exec or a volume. Only stdout and stderr are collected.
• Rotation: The kubelet rotates container log files on the node. A logging agent (DaemonSet) ships them off the node if you need history after the Pod is gone.

Where the files sit on the node

When the API server cannot serve logs (kubelet down, Pod never registered), read the node directly.

crictl pods
crictl ps -a
crictl logs <container-id>
journalctl -u kubelet --since "15 min ago" --no-pager

crictl uses /etc/crictl.yaml so it talks to the same runtime socket as the kubelet (containerd’s socket is typically /run/containerd/containerd.sock).

Control plane logs

On kubeadm, apiserver, scheduler, controller-manager, and etcd are static Pods. Their stdout is available like any other Pod when the API is up:

kubectl logs -n kube-system etcd-controlplane
kubectl logs -n kube-system kube-apiserver-controlplane
kubectl logs -n kube-system kube-scheduler-controlplane --tail=100

The node hostname is the suffix. List them with kubectl get pods -n kube-system -o wide.

If the API server is the thing that is down, use the node:

crictl ps -a | head
crictl logs <apiserver-container-id>
ls /var/log/pods | grep kube-apiserver

etcd also logs to its container stdout. Disk and member health are a separate check. On the control-plane node:

sudo ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
df -h /var/lib/etcd

Nospc in the etcd log means the member is out of disk and will not elect until space is freed. Snapshot with etcdctl snapshot save /opt/etcd-backup.db and the same flags before you change the data directory.

metrics-server and kubectl top

• metrics-server is a cluster addon. It scrapes the kubelet Summary API and exposes the Metrics API (metrics.k8s.io).
• kubectl top reads that API. Without metrics-server the command errors with “Metrics API not available.”
• HorizontalPodAutoscaler reads this same Metrics API. kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10 scales a Deployment from CPU utilization, which is current usage divided by the container’s CPU request. Pods with no CPU request cannot scale on utilization. Until kubectl top pods works, an HPA has nothing to scale on.

kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl top pod nginx --containers
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get deploy metrics-server -n kube-system
kubectl logs -n kube-system deploy/metrics-server

• Typical lab failure: metrics-server cannot verify the kubelet TLS certificate. The Deployment args then include --kubelet-insecure-tls. Check the Pod logs before you assume the addon is missing.
• Requests vs usage: kubectl describe node shows requests (what scheduling reserved). kubectl top shows usage (what is happening now). A node can be full of requests and still look idle in top.

Node conditions that show up next to metrics

kubectl describe node reports conditions the scheduler and eviction manager honor:

Under pressure the kubelet evicts Pods by QoS class, which is derived from requests and limits:

A short triage order

1. kubectl get pods -o wide — phase, restarts, node.
2. kubectl describe pod — Events at the bottom (Failed, Unhealthy, FailedScheduling, FailedMount).
3. kubectl logs and, if restarts > 0, kubectl logs --previous.
4. kubectl top pod when the symptom is CPU, memory, or HPA.
5. On the node: journalctl -u kubelet and crictl logs when the API cannot answer.

Official docs

• kubectl logs
• Metrics server
• Troubleshooting applications
• crictl