Networking

Networking Quick Notes

The model

Three rules the cluster is built to honor:

1. Every Pod gets its own IP. Containers in the Pod share that IP.
2. Pods can reach other Pods without NAT, on any node.
3. Nodes and Pods can reach each other without NAT. Agents on the node (kubelet) call Pod IPs directly.

Service ClusterIPs are an extra virtual layer on top of this. They are not Pod IPs.

Two CIDRs you will be asked to tell apart:

The kubernetes Service in default takes the first address of the Service CIDR (commonly 10.96.0.1) and is how Pods reach the API server.

CNI

Container Network Interface is how the kubelet asks a plugin to attach a network namespace and an IP when a Pod starts.

• Config: /etc/cni/net.d/*.conf (or .conflist).
• Binaries: /opt/cni/bin (bridge, host-local, loopback, portmap, plus the vendor plugin).
• Who calls it: The kubelet, on PodSandbox create and teardown.
• After kubeadm init: The node stays NotReady until a CNI addon is installed. CoreDNS cannot become Ready until Pod sandboxes get addresses.

Common plugins and the reason you would pick one:

Install is “apply the manifest the vendor ships,” with the Pod CIDR matching kubeadm init --pod-network-cidr. A mismatch (plugin using 10.244.0.0/16 while kubeadm was given 192.168.0.0/16) produces nodes that look Ready and Pods that cannot route.

Check:

ls /etc/cni/net.d
ls /opt/cni/bin
kubectl get pods -n kube-system -o wide | grep -E 'calico|cilium|flannel|weave'
ip addr show

The Pod veth and the bridge or tunnel interface exist on the node only after the plugin has programmed them. crictl pods shows the sandbox. Absence of a CNI config is the usual cause of NetworkNotReady / “plugin type not found” in journalctl -u kubelet.

kube-proxy and Services

kube-proxy watches Services and EndpointSlices and programs the node so a packet to a ClusterIP is DNATed to a ready Pod IP.

• Modes: iptables (default on many clusters), ipvs (scales to more Services), or an nftables mode on recent clusters. Cilium’s kube-proxy replacement does the same job in eBPF.
• Ready only: Pods that fail readiness are removed from the EndpointSlice. kube-proxy stops sending them traffic. The ClusterIP still answers; it just has fewer backends.
• No endpoints: The ClusterIP blackholes. Selector typo and “Pod labels do not match the Service” are the first things to compare.

kubectl get svc web -o wide
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl get pods --show-labels

EndpointSlices replaced the older Endpoints object as the API kube-proxy scales on. kubectl get endpoints still works for a quick look. If the slice is empty, fix labels or readiness before you touch the CNI.

kube-proxy itself is usually a DaemonSet in kube-system. If those Pods are down, ClusterIP from other Pods fails while Pod-IP-to-Pod-IP traffic (pure CNI) still works. That split is the fastest way to tell “CNI is broken” from “kube-proxy is broken.”

port is the Service port. targetPort is the container port. A headless Service sets clusterIP: None and DNS returns Pod IPs. kube-proxy programs the forwarding. It only includes Pods that are Ready and whose labels match the Service selector.

Pod-to-Pod path

Same node: the CNI bridge (or veth pair) forwards directly.

Across nodes: the plugin encapsulates (VXLAN, Geneve) or routes (BGP, cloud routes). The node whose Pod CIDR slice contains the destination delivers the packet. ip route on the node shows the slice this node owns.

hostNetwork: true Pods use the node IP and the node namespace. They do not get a Pod CIDR address. Ports collide with other host-network Pods and with the node itself.

CoreDNS

CoreDNS runs as a Deployment (or DaemonSet) in kube-system and is exposed as the kube-dns Service. Every Pod’s /etc/resolv.conf points at that Service IP, with a search list:

search <namespace>.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

ndots:5 means a name with fewer than five dots is tried against each search suffix first. web in namespace shop becomes web.shop.svc.cluster.local.

Named ports also get SRV records (_<port>._<proto>.<service>.<ns>.svc.cluster.local). Clients that only need the Service use the A record. The name you want in an app is the Service FQDN.

The Corefile lives in the coredns ConfigMap in kube-system. The kubernetes plugin zone must match the cluster domain (cluster.local by default). A bad Corefile crash-loops CoreDNS and every name in the cluster fails, including the apiserver name from inside Pods.

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl get configmap coredns -n kube-system -o yaml
kubectl exec -it debug -- nslookup kubernetes.default.svc.cluster.local

Stub domains and forwarding (forward . /etc/resolv.conf) are how cluster DNS reaches names outside the cluster. If Pods resolve kubernetes.default and fail on public names, the upstream forward is the piece to fix.

Ingress

An Ingress object describes HTTP(S) routes. An Ingress controller (a Deployment you install: ingress-nginx, Traefik, HAProxy, a cloud controller) is the process that reads those objects and configures a real proxy. Without a controller, kubectl get ingress shows an address that stays empty and no traffic moves.

Current API is networking.k8s.io/v1.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: store
  namespace: shop
spec:
  ingressClassName: nginx
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /wear
        pathType: Prefix
        backend:
          service:
            name: wear
            port:
              number: 80
      - path: /watch
        pathType: Prefix
        backend:
          service:
            name: watch
            port:
              number: 80
  tls:
  - hosts: ["shop.example.com"]
    secretName: shop-tls

• ingressClassName selects which controller handles this object. The controller’s IngressClass is a cluster-scoped object. One class may be default (ingressclass.kubernetes.io/is-default-class: "true").
• pathType: Prefix matches /wear and /wear/coat. Exact matches only that path. ImplementationSpecific is left to the controller.
• Host rules send shop.example.com and api.example.com to different Services. A rule with no host matches any host that reached the controller.
• TLS uses a Secret of type kubernetes.io/tls in the same namespace as the Ingress.
• The backend is a Service, almost always ClusterIP. The controller Pods must be able to reach that Service (NetworkPolicy included).
• Publish the controller with a Service of type NodePort or LoadBalancer, or with hostNetwork. Creating only the Ingress resource does not open a port on the node.

nginx ingress rewrite (the annotation labs still use):

metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /

That annotation is controller-specific. It is not a field in the Ingress API. kubectl describe ingress store shows the controller’s events (Sync, Create from the nginx class).

kubectl get ingressclass
kubectl get ingress -A
kubectl describe ingress store -n shop

Gateway API

Gateway API splits the same job across roles. Ingress keeps the route and the listener in one object. Gateway API separates them.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: store
  namespace: shop
spec:
  parentRefs:
  - name: public
    namespace: edge
  hostnames: ["shop.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /wear
    backendRefs:
    - name: wear
      port: 80

• parentRefs attaches the route to a Gateway. The route does nothing until a Gateway accepts it (Accepted in status.parents).
• Cross-namespace backends need a ReferenceGrant in the Service’s namespace.
• Ingress and Gateway API can both be installed. ingressClassName does not select a Gateway, and parentRefs does not select an Ingress controller.

NetworkPolicy, from the packet’s point of view

Policy is enforced by the CNI on the Pod’s interfaces, for Pod IPs. A Service VIP is still how clients connect. The allow rules must match the source Pod (or an ipBlock) and the destination port on the backend. With no policies in a namespace, all ingress and all egress are allowed. Once a Pod is selected, traffic of a listed policyTypes entry is denied unless a rule allows it. An empty ingress: [] denies all ingress. Egress stays open until a selected policy lists Egress. Forgetting UDP/TCP 53 in an egress policy breaks DNS.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-allow-api
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 80

podSelector inside from, with no namespaceSelector, means Pods in this namespace. A namespaceSelector alone means all Pods in matching namespaces. Both in one from item are ANDed. Separate from items are ORed. podSelector: {} on the policy selects every Pod in the namespace. Flannel does not enforce NetworkPolicy. Calico and Cilium do. If the plugin ignores the object, the YAML applies and traffic still flows.

If wget to the Pod IP fails and wget to a Pod in another namespace works, suspect policy or the routing table, then CoreDNS only if the failure is “bad address” rather than a timeout.

Official docs

• Cluster networking
• Services
• DNS for Services and Pods
• Ingress
• Gateway API