Cluster Design

Cluster Design Quick Notes

What you are sizing

The Pod CIDR cannot be renamed casually after init. Service CIDR neither. Pick them so they do not overlap the VPC, the office, or each other.

Managed and self-managed

With kubeadm you bring the machines. kubeadm init writes certificates, static Pod manifests under /etc/kubernetes/manifests, and admin.conf. Nodes stay NotReady until you install a CNI whose Pod CIDR matches --pod-network-cidr. Workers join with kubeadm join and a short-lived token. You still choose how many control planes, whether etcd is stacked or external, and which addon provides CNI, CSI, and Ingress.

High availability

One control plane is a single failure domain. HA means:

• At least three control-plane nodes (or three stacked etcd members).
• One stable address in front of the apiservers: a TCP load balancer or a DNS name with a short TTL. kubeadm calls this --control-plane-endpoint (for example cp.example.com:6443). Every kubeconfig and every join uses that name, not a single node’s private IP.
• Loss of one member leaves the cluster able to elect an etcd leader and serve API writes.

Workers are already multiple. Adding workers does not make the control plane highly available.

kubeadm taints control-plane nodes node-role.kubernetes.io/control-plane:NoSchedule. Leave the taint on so workloads stay on workers. Workloads on the control plane compete with etcd for disk and CPU. Remove it only when a single node must run both.

Stacked etcd and external etcd

Stacked: etcd runs as a static Pod on each control-plane node, next to the apiserver. kubeadm’s default.

• Fewer machines (three nodes can be a complete HA control plane).
• A node loss removes an apiserver and an etcd member together.
• Couples etcd disk IO to whatever else that node does.

External: etcd runs on a separate set of machines. Control planes only run apiserver, scheduler, and controller-manager, and point --etcd-servers at the etcd cluster.

• etcd can be sized and snapshotted on its own.
• You operate two sets of machines and two failure domains.
• Member count is still odd and independent of the apiserver count (three etcd, two or more apiservers is a valid layout).

etcd quorum

etcd commits a write when a majority of members agree.

An even count (2 or 4) does not tolerate more failures than the odd count below it, and it costs another machine. Add members one at a time so the cluster always has a leader. Replacing a member is etcdctl member add / member remove, then a new peer with the matching --initial-cluster data.

A single-member snapshot and restore, run on the control plane with the etcd client certs in /etc/kubernetes/pki/etcd/:

sudo ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

sudo ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore

snapshot restore does not overwrite the live directory. Edit /etc/kubernetes/manifests/etcd.yaml so the data volume hostPath and --data-dir use /var/lib/etcd-restore. The kubelet restarts the etcd static Pod. The apiserver stays down until etcd answers again.

Extension interfaces

Kubernetes stays small by calling out:

The core binaries do not contain Calico, EBS, or ingress-nginx. Those are addons you install after kubeadm init.

CRDs and operators

A CustomResourceDefinition adds a kind to the API (certificates.cert-manager.io, grafanaagents.grafana.com). After the CRD is established, kubectl get <plural> works like any built-in.

kubectl get crd
kubectl explain certificate --api-version=cert-manager.io/v1

An operator is a controller that watches those objects and reconciles something outside the object (issue a certificate, deploy Prometheus, resize a database). Installing an operator is usually “apply a CRD, apply a Deployment, then create a custom resource.” If the CRD is missing, the custom resource returns “no matches for kind”. If the controller Deployment is down, the object exists and its status never becomes Ready.

Helm

Helm installs a chart (templated manifests) as a named release.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace
helm list -A
helm upgrade ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx
helm uninstall ingress-nginx -n ingress-nginx
helm get manifest ingress-nginx -n ingress-nginx

• Release name is how you upgrade and delete. The generated object names usually include it.
• Values (-f values.yaml or --set key=value) are the supported way to change a chart. Editing a live object by hand is overwritten on the next helm upgrade.
• Namespace is per release. helm list without -A hides other namespaces.
• Charts vendor their own CRDs. Install order problems (“no matches for kind”) usually mean the CRDs have not been applied yet. helm install applies CRDs from the chart’s crds/ directory first. helm upgrade does not upgrade those CRDs; apply CRD changes on purpose.

Kustomize

Kustomize patches plain YAML without a templating language. kubectl has it built in.

kustomization.yaml
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: web
images:
  - name: nginx
    newTag: "1.27"

kubectl apply -k ./overlays/prod
kubectl kustomize ./overlays/prod | less

• resources: Files or directories to include.
• commonLabels / labels: Added to every object and, where it matters, to selectors. Changing a selector later orphans Pods. Prefer a label that is stable.
• images.newName / newTag: Rewrite image references at apply time.
• patches: Strategic merge or JSON patch against one object.
• Bases and overlays: overlays/prod has a kustomization.yaml whose resources point at ../../base. Prod and dev differ by patches, not by forked copies of the Deployment.

kubectl apply -f on a directory does not run Kustomize. The flag is -k.

Choosing an install tool

Official docs

• Highly available clusters with kubeadm
• Custom resources
• Helm docs
• Kustomize