Kubernetes Networking Faults: CNI, Services, CoreDNS and How to Trace a Packet

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A Kubernetes connection fault normally sits in one of four layers: name resolution, the Service, the pod network or policy. Test by name, then by Service IP, then by pod IP, then across nodes. Comparing those tests narrows the likely layer; correctly placed packet captures then help locate the failure.

The commands below were not run in a lab. Names and addresses are examples.

Recommendations and interpretations are editorial unless attributed; source research is not lab validation.

Why it matters

"The network is down" covers several separate systems that fail in different ways. Kubernetes itself implements only a few parts of its network model. The pod network comes from a plugin, Services from a proxy, names from an add-on, and policy usually from the plugin again.

Changing any of them on a guess can cause a second fault and destroy the evidence.

How it works

The four parts

The container runtime, not the kubelet, loads the CNI plugins. The kubelet's own CNI flags were removed in Kubernetes 1.24, so old troubleshooting notes that mention them are out of date.

The CNI specification is narrow. CNI sets up the pod's interface. Everything after that belongs to the plugin's own data plane.

What happens to a Service connection

In the default proxy mode, a connection to a ClusterIP is matched by a rule for that Service, and a backend is chosen by session affinity or at random. Destination NAT then rewrites the destination to that pod's IP, usually preserving the client's address on an ordinary pod-to-Service path; hairpin and other masquerading cases can differ.

Two practical results follow. A ClusterIP exists only as rules for its ports, so test it with the real port and not with ping. And on the wire between nodes you will see pod IPs, not the Service IP.

Know which proxy you have, because each one is inspected differently:

On a node, curl http://localhost:10249/proxyMode reports the kube-proxy mode. If the cluster does not use kube-proxy, the iptables and IPVS checks do not apply.

The fault-finding order

Start by recording the facts: source pod, destination, port, and the node each runs on.

kubectl get pod -n app -o wide

Then run these tests from the failing client pod, in order. This is our arrangement of the steps in the Kubernetes Service debugging guide.

If step 3 fails from everywhere, test the application inside its own pod. Then review NetworkPolicy, because a policy can produce a failure at steps 1 to 4 and mimic any other layer.

Keep the same client, protocol, port and application identity for steps 1 to 3. HTTP Host headers and TLS server names can change behaviour when testing an IP directly; preserve them before attributing a difference to DNS.

DNS

Work from the client outwards, as the DNS debugging guide does:

kubectl exec -ti client -n app -- cat /etc/resolv.conf
kubectl exec -ti client -n app -- nslookup kubernetes.default
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns

If the client image has no nslookup, the guide supplies a dnsutils test pod. The CoreDNS Service and pods are still named and labelled kube-dns for backward compatibility. nameserver must point at the cluster DNS Service, and search must include the namespace suffixes.

If a fully qualified name works but a short one fails, the search path is the problem. If kubernetes.default fails, DNS itself or the path to it is broken.

To confirm queries are arriving, add log to the Corefile in the coredns ConfigMap. Changes can take up to a minute or two to propagate when the Corefile includes the reload plugin, as the guide's example does. This edits a production ConfigMap in kube-system, so treat it as a change. Remove it when you finish, because logging every query on a busy cluster is noisy.

Known causes, each with a specific signature:

• CoreDNS in CrashLoopBackOff with a Loop ... detected message. The loop plugin found a forwarding loop, usually a node /etc/resolv.conf that lists 127.0.0.53 from systemd-resolved. The documented fix is the kubelet resolvConf setting pointing at the real file. On systems using systemd-resolved, that is typically /run/systemd/resolve/resolv.conf.
• SERVFAIL for cluster names. CoreDNS must be able to list Service and EndpointSlice resources. Check kubectl describe clusterrole system:coredns.
• DNS stops the moment a policy is applied. A default-deny egress policy also blocks DNS, so add a policy that allows it.
• Large answers fail on older images. Alpine 3.17 and earlier lack TCP fallback, so DNS responses above 512 bytes fail.
• Upstream servers lost. glibc allows three nameservers by default and Kubernetes consumes one. The guide's workarounds are dnsmasq on the node or the kubelet --resolv-conf flag.
• REFUSED under load. If the forward plugin has max_concurrent set, queries above the limit are refused.

Why external names look slow

Pods get ndots:5 by default. A name with fewer dots than that is tried against the search list before being tried as an absolute name.

So a lookup of www.example.com, with two dots, first tries the cluster suffixes and collects a series of failures before the real query (our derivation). A trailing dot makes the name absolute. A pod's dnsConfig can lower ndots, but short service names may then stop resolving, so test that.

Intermittent timeouts have a separate cause. UDP DNS through a Service uses connection tracking, and Kubernetes documents conntrack races and UDP entries filling the table. It also describes timeouts of up to 30 seconds from dropped UDP packets. NodeLocal DNSCache is the documented design response. It skips DNAT and connection tracking for pod queries and can use TCP upstream. Those causes apply only where queries pass through kernel connection tracking, and two proxies change the install. In IPVS mode the cache cannot bind the CoreDNS cluster IP, so the kubelet's --cluster-dns must point at the local address. With Cilium handling Services, the cache's interception is bypassed, and Cilium gives its own steps using a Local Redirect Policy.

Services

If the Service IP fails, check the object first:

kubectl get svc api -n app -o yaml
kubectl get endpointslices -n app -l kubernetes.io/service-name=api

If the endpoints are <none>, the selector does not match the pod labels. Also confirm that targetPort matches the port the container actually listens on.

A pod that is running but not ready is easy to miss. The EndpointSlice ready condition means serving and not terminating, so a failing readiness probe removes the pod from normal Service traffic.

Traffic policy can make a healthy Service look broken on some nodes:

• With internalTrafficPolicy: Local, a client on a node without a local endpoint sees a Service with no endpoints.
• With externalTrafficPolicy: Local, packets sent to a node without a local endpoint are dropped. The same setting preserves the client source IP, which the default NodePort path replaces through SNAT.

If a pod cannot reach itself through its own Service, consider hairpin mode on the kubelet, usually with kube-proxy in iptables mode and pods on a bridge network. The flag must be hairpin-veth or promiscuous-bridge.

For nftables mode, note two differences from iptables. NodePorts listen only on the node's primary addresses by default, and kube-proxy adds no firewall accept rules, so a local firewall must allow the NodePort range. Inbound NodePort ports are 30000-32767 by default.

The same page warns that Linux kernels before 6.1 have a bug that can close long-lived TCP connections to Service IPs with Connection reset by peer. iptables mode installs a workaround that was later found to cause other problems, and nftables mode does not by default, so check the kernel before moving. The iptables_ct_state_invalid_dropped_packets_total metric shows whether you depend on the workaround, and --conntrack-tcp-be-liberal provides it in nftables mode.

NetworkPolicy

Policy is enforced only if the plugin implements it. Without that the objects exist and have no effect, so a policy that "should" block traffic may do nothing.

Where it is enforced, policies are additive and a connection needs both the source's egress and the destination's ingress to allow it. Traffic between a pod and its own node is always allowed, so a test from the node proves nothing about policy.

kubectl get networkpolicy -A

The pod network

Pods stuck during creation, or failing to get an address, point at the CNI layer. Read the pod's events and the runtime log first. For containerd 2.3 the defaults are plugins in /opt/cni/bin and configuration in /etc/cni/net.d, loading one configuration file. So check for a missing or unreadable file in that directory and for a crashed plugin agent on the node, which are the usual current causes.

A legacy case may still be in old runbooks. Kubernetes documents two version-mismatch errors, incompatible CNI versions and failed to destroy network for sandbox, tied to containerd v1.6.0 to v1.6.3. Containerd 1.6 reached end of life on 23 August 2025, and Kubernetes 1.37 pairs with containerd 2.3 or later, so a current cluster is unlikely to show them.

If pods work within a node but not across nodes, look at encapsulation or routing. Check that the underlying network permits these, as documented by each project:

Calico says that when containers reach each other on the same host and reach the Internet, but not containers on different hosts, suspect BGP. Run calicoctl node status and look for Established. That applies only where BGP is the networking mode, so it will not explain a VXLAN install.

When large packets vanish

If small requests work and large transfers hang, suspect MTU. Encapsulation costs space: 50 bytes for IPv4 VXLAN, 20 for IP in IP, 60 for IPv4 WireGuard. So a 1500-byte network carries a pod MTU of 1450 with VXLAN, 1480 with IP in IP and 1440 with WireGuard. Calico auto-detects the MTU by default, and says to disable unused encapsulations so detection can pick the right value. On a default install the question is why it chose that value, and a leftover encapsulation setting is the first thing to check. The VXLAN standard says tunnel endpoints must not fragment VXLAN packets, and a receiving endpoint may silently discard fragments, which fits a loss with no error.

Read the real values first, because the fault may be a wrong MTU. A figure from the node can mislead too: on AKS the underlying network has an MTU of 1400, even though the interface shows 1500.

kubectl exec client -n app -- cat /sys/class/net/eth0/mtu

Then test with a size that fills that MTU and with fragmentation prohibited. -M do sets the DF flag and -s sets the data bytes. Those flags belong to the iputils ping, and a minimal image may have a different one or none. The data size is the pod MTU less 28 bytes of IPv4 and ICMP headers (our arithmetic), so 1472 if the file says 1500 and 1422 if it says 1450:

kubectl exec -ti client -n app -- ping -M do -s <pod-mtu-minus-28> -c 3 <pod-ip-on-other-node>

A local message too long error means the size exceeds the sender's own interface. Silence at full size, with replies at a much smaller size, points at the path. If the small size also fails, MTU is not shown at all, because a NetworkPolicy may deny ICMP. Then measure the path between nodes the same way. Note that a changed MTU in Calico applies only to new workloads.

Tracing the packet

When the layer is known but not the place, capture at each hop. Kubernetes documents this approach using a debug pod. The step below pulls an image and installs packages, which may fail while DNS or egress is down, so mirror a debug image containing tcpdump and conntrack inside your environment beforehand.

kubectl debug --profile=sysadmin node/<node> -it --image=ubuntu:latest
# inside the debug container
apt-get update && apt-get install -y tcpdump
tcpdump -i any -n 'host <client-pod-ip> and (port <service-port> or port <target-port>)'

Start with the client address and expected ports, accounting for any source translation and all eligible backends. To see inside a pod, use the pod form, kubectl debug --profile=sysadmin pod/<pod>, which captures in the pod's network namespace. A node capture sees only the host side of the pod's interface.

Warning: the sysadmin profile grants root-level access on the node, and captures can contain credentials and customer data. Treat this as a change on production, keep captures short and delete them afterwards. Delete the debug pod when finished.

-i any captures from all regular interfaces and -n stops address lookups, so the same packet can show at several points on a node, before and after translation. Read the sequence like this. This applies to kube-proxy only, and on an overlay the pod addresses are hidden inside the tunnel on the physical interface:

To see the tunnel itself, capture on the node's physical interface with the port from the table above, for example tcpdump -i eth0 -n udp port 8472.

With kube-proxy, connection tracking shows the translation directly, if the tool is installed in your debug container. conntrack -L lists entries, and -d filters on the original destination:

conntrack -L -d <service-ip>

The reply direction of the entry shows which backend was chosen.

Cilium's replacement depends on the socket-LB feature, where the kernel socket connects straight to the backend and no lower-layer NAT is needed. So a pod's capture may show no Service IP and conntrack has no reason to show one, and the table above does not apply. Packets handled by NodePort XDP are not visible in tcpdump either.

Use Cilium's own tools. hubble observe --pod <pod> --verdict DROPPED lists dropped flows, provided Hubble is enabled, its CLI is installed and its API is accessible. cilium-dbg monitor --type drop shows where and why packets are dropped, such as Policy denied. Run cilium-dbg inside the agent, on the node concerned, because ds/cilium selects only one agent:

kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system exec <agent-on-that-node> -- cilium-dbg monitor --type drop

What to choose and why

The following are our recommendations:

• Set the kube-proxy mode explicitly. Kubernetes advises this so the backend does not change during an upgrade.
• Where queries pass through connection tracking, add NodeLocal DNSCache before DNS volume makes intermittent timeouts a recurring incident, using the IPVS or Cilium steps if those apply.

Risks and limits

• Do not flush or hand-edit proxy rules. kube-proxy owns them, and a manual change hides the original state.
• Restarting components can erase the fault. Capture evidence first.
• Do not widen policy to test. A temporary allow-all can expose workloads and proves only that a policy exists somewhere.
• Not covered: cloud provider load balancers, Ingress and Gateway implementations, service meshes, Windows nodes and IPv6-specific behaviour. Sidecars and mesh proxies add their own hops, so check their documentation as well.

Sources

Opened 2026-11-10.

• Kubernetes: Networking concepts, Network plugins, Virtual IPs, Debug Services, Debugging DNS, DNS for Services and Pods, NodeLocal DNSCache, EndpointSlices, NetworkPolicy, Internal traffic policy, Source IP, Ports and protocols, Debug running pods, CNI errors
• CNI specification, containerd CRI configuration, RFC 7348 VXLAN
• CoreDNS: loop, forward
• Calico: Requirements, MTU, Troubleshooting. Cilium: Routing, Kube-proxy replacement, Hubble CLI, Troubleshooting
• Manual pages: resolv.conf(5), hostname(7), ping(8), tcpdump(1), conntrack(8), nft(8)
• Cilium: Local Redirect Policy, containerd releases

This article was originally published on 10 November 2026.