Scheduling

Scheduling Quick Notes

What the scheduler does

• Input: A Pod whose spec.nodeName is empty.
• Filter: Drop nodes that cannot run it (taints, insufficient CPU/memory, node affinity, unbound volumes, disk pressure).
• Score: Rank the remaining nodes (spread, requested resources, preferred affinity).
• Bind: Set spec.nodeName through the API. The kubelet on that node starts the containers.
• Pending forever: If no node passes the filters, the Pod stays Pending. kubectl describe pod shows the FailedScheduling events.

Manual scheduling

• spec.nodeName: You name the node. The scheduler skips the Pod.
• Use when: You must pin a Pod to a node you already know, such as node01.
• Constraint: The node must already exist. If it does not, the Pod stays Pending forever. Setting nodeName skips scheduler filters, including NoSchedule taints, so use it only for an explicit pin. The kubelet can still fail the start (bad image, missing volume).

spec:
  nodeName: node01
  containers:
  - name: nginx
    image: nginx

A Pod that is already running keeps its node. To move it, delete it and let the controller recreate it (or set nodeName on a new object).

Labels and selectors

• Labels: Indexed keys on any object. app=nginx, tier=frontend, disk=ssd.
• Equality selector: app=nginx, tier!=cache.
• Set selector: env in (prod, stage), env notin (dev), disk (key exists), !gpu.
• Who uses them: Services, ReplicaSets, Deployments, NetworkPolicies, node selectors, and kubectl -l.

kubectl label node node01 disktype=ssd
kubectl get pods -l 'env in (prod,stage),tier=front'
kubectl get pods -L app,env

Annotations are the wrong tool for selection. The scheduler and Services do not match on annotations.

Taints and tolerations

A taint is the node saying “stay off unless you tolerate this.”

kubectl taint nodes node01 dedicated=batch:NoSchedule
kubectl taint nodes node01 dedicated=batch:NoSchedule-    # remove
kubectl describe node node01 | sed -n '/Taints/,+5p'

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: batch
    effect: NoSchedule

• operator: Exists: Match the key and effect, any value. Omit value.
• tolerationSeconds: With NoExecute, how long the Pod may linger before eviction.
• Control plane: kubeadm taints control-plane nodes with node-role.kubernetes.io/control-plane:NoSchedule. Workloads land on workers. Static Pods for apiserver/etcd are started by the local kubelet and are not scheduled in the normal way.
• Built-in eviction taints: node.kubernetes.io/not-ready and node.kubernetes.io/unreachable are NoExecute. The node controller adds them. Default tolerations on Pods give a short grace period.

A toleration allows a Pod onto a tainted node. It does not force the Pod onto that node. To force placement, combine the toleration with nodeSelector or a required node affinity.

nodeSelector

• Hard equality against node labels.
• Simple and enough when the rule is equality on one label, such as “only nodes with disktype=ssd”.

spec:
  nodeSelector:
    disktype: ssd

If no node has that label, the Pod stays Pending.

Node affinity

Richer than nodeSelector. Rules use the same operators as set selectors (In, NotIn, Exists, DoesNotExist, Gt, Lt).

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values: ["ssd"]
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values: ["a"]

• nodeSelectorTerms are ORed. Multiple matchExpressions inside one term are ANDed.
• Pod affinity / pod anti-affinity use the same required/preferred pattern against labels of other Pods (topologyKey such as kubernetes.io/hostname or topology.kubernetes.io/zone). Anti-affinity spreads replicas. The fields sit in the same spec.affinity block as node affinity.

Taints vs affinity

Use both when a node is reserved for a workload: taint the node so strangers stay off, and add a required affinity so the workload actually lands there.

Requests and limits

CPU is written as cores: 100m = 0.1 core, 1 = 1 core. Memory uses binary units: 128Mi, 1Gi.

QoS class is derived, not set by you:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "256Mi"

That Pod is Burstable. Set request and limit to the same values on every container for Guaranteed.

LimitRange in a namespace can default requests and limits. ResourceQuota caps the namespace total. A Pod that asks for more than the quota stays Pending.

DaemonSets

• One Pod per matching node, including nodes that join later.
• Typical members: kube-proxy, a CNI agent, a log collector, a node exporter.
• Node selection: nodeSelector or node affinity on the template. A DaemonSet tolerates common node taints only if you put those tolerations in the template. Control-plane nodes are skipped unless the template tolerates node-role.kubernetes.io/control-plane:NoSchedule.
• Update: RollingUpdate with maxUnavailable, or OnDelete.

kubectl get ds -A
kubectl describe ds kube-proxy -n kube-system

Static Pods

• Defined by a file the kubelet watches, default directory /etc/kubernetes/manifests (staticPodPath in the kubelet config).
• The API server is not required to start them. The kubelet reads the file and calls the runtime.
• Mirror Pod: The kubelet also creates a read-only Pod object in the API so kubectl get pods -n kube-system shows it. The name is <pod>-<node-hostname>, for example etcd-controlplane.
• You cannot delete the mirror from the API and expect it to stay gone. The kubelet recreates it. Remove or fix the manifest file on the node.
• kubeadm control plane: apiserver, etcd, scheduler, and controller-manager are static Pods. A broken apiserver is often a bad flag or volume in /etc/kubernetes/manifests/kube-apiserver.yaml. After you edit the file, the kubelet restarts the Pod. If the API is down, on that node run crictl ps -a, crictl logs <container-id>, and journalctl -u kubelet --since "20 min ago". YAML errors and unknown flags show up there. Move the manifest out of the directory and back if you need to force a restart without deleting the definition.

Multiple schedulers

• spec.schedulerName: Defaults to default-scheduler.
• A second scheduler is a Deployment (or static Pod) running kube-scheduler with its own config and a different --scheduler-name / leader-elect lock.
• Pods that name it are ignored by the default scheduler and placed by the custom one.
• If that scheduler is down, those Pods stay Pending even when the default scheduler is healthy.

spec:
  schedulerName: custom-scheduler

kubectl get pods -A -o custom-columns=NAME:.metadata.name,NS:.metadata.namespace,SCHED:.spec.schedulerName

Official docs

• Assigning Pods to nodes
• Taints and tolerations
• Resource management
• DaemonSet
• Static Pods