About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
A node that shows NotReady, with conditions that read Kubelet stopped posting node status., has stopped sending status and heartbeats to the control plane. The node controller writes that message. It says contact was lost, not why. Check on the node whether the kubelet and the container runtime are running, whether the node can reach the API server, and whether its kubelet client certificate is valid.
What the symptom means
A node reports in two ways. The kubelet updates the node's .status, and it renews a Lease object in the kube-node-lease namespace. The node status page says the kubelet renews that Lease every 10 seconds by default, independently of status updates.
The Ready condition has three values. The node status page says it is True if the node is healthy, False if it is not healthy and not accepting pods, and Unknown "if the node controller has not heard from the node in the last node-monitor-grace-period (default is 50 seconds)". The same page's heartbeat section describes a 40 second default timeout for unreachable nodes, so read the value your cluster actually uses in the kube-controller-manager settings.
The text comes from the node lifecycle controller's source code (release tag v1.34.0), where a condition that has not been updated is set to Unknown with the reason NodeStatusUnknown and the message Kubelet stopped posting node status. The troubleshooting page shows the same text on the Ready, MemoryPressure, DiskPressure and PIDPressure conditions of a node that is down.
After the grace period the control plane adds a node.kubernetes.io/unreachable taint for Unknown, or node.kubernetes.io/not-ready for False. The nodes page says that by default the node controller waits 5 minutes between marking the node Unknown and submitting the first eviction request for its pods.
Likely causes
| Cause | What you see | How to confirm |
|---|---|---|
| The kubelet has stopped or crashed | Node Unknown; the machine still answers SSH | systemctl is-active kubelet, then the journal |
| The container runtime is down | The kubelet runs but cannot start or report containers | crictl ps returns an error or does not answer on the node |
| The machine is off, hung or cut off from the network | Node Unknown; no SSH, no ping | Hypervisor or console view; network path to the API server |
| The kubelet's client certificate expired or failed to rotate | The kubelet runs, yet the control plane rejects it | x509: certificate has expired or is not yet valid in kube-apiserver logs |
| The node is out of disk, memory or process IDs | Conditions DiskPressure, MemoryPressure or PIDPressure True | kubectl describe node conditions |
| The node cannot reach the API server | The kubelet journal shows connection errors to the API address | Test from the node to the API server address and port |
Prove the cause before changing anything
All of these are read-only.
1. Read the node's conditions, taints and lease.
kubectl get nodes
kubectl describe node k8s-node2
The troubleshooting page shows a down node with Conditions that carry NodeStatusUnknown and Kubelet stopped posting node status., a Taints line of node.kubernetes.io/unreachable:NoExecute, and a Lease whose RenewTime is earlier than the conditions' LastTransitionTime. A good result on a healthy node is Ready True. The node status page's healthy example shows reason KubeletReady and message kubelet is posting ready status.
2. Check the kubelet service on the node, if you can still log in.
systemctl is-active kubelet
systemctl status kubelet
The systemd manual says is-active returns an exit code 0 if a unit is active, and prints the current state. status shows runtime status followed by recent journal lines.
3. Read the kubelet's recent log.
journalctl -u kubelet -n 100 --no-pager
The journalctl manual describes -u as showing messages for a unit and -n as limiting the number of events shown. Look for repeated errors about the API server address, certificates or the runtime. The debugging page for clusters says that on systemd-based systems you may need journalctl instead of log files.
4. Check the container runtime.
crictl ps
The crictl page lists running containers with crictl ps. If it returns an error or does not answer, the runtime is the next thing to check; the page itself does not say so.
5. If you cannot log in, use the hypervisor or console. The node debugging page says kubectl debug node does not work when the node is down (disconnected from the network, or the kubelet dies and will not restart), so it is not a way in for this fault.
6. Check the kubelet certificate. If the log or the kube-apiserver log shows x509: certificate has expired or is not yet valid, the case is covered in our guide to expired kubeadm certificates and by the kubeadm troubleshooting page. Note that kubeadm certs check-expiration does not list kubelet.conf.
Fix
Fix the cause found. The kubelet's certificate case follows the kubeadm troubleshooting steps named in check 6.
The kubelet or the runtime service has stopped
First save the status of the unit you will restart and its recent log. The unit is kubelet unless the checks showed that the container runtime's unit has stopped; write that unit's name in place of <unit>:
systemctl status <unit> > /root/<unit>-status-2026-11-22.txt
journalctl -u <unit> -n 200 --no-pager > /root/<unit>-journal-2026-11-22.txt
Disruptive step: the restart command below restarts the kubelet on this node, which stops the kubelet and starts it again. If you restart the container runtime's unit instead, its containers can be interrupted. Kubernetes documentation gives no guarantee that running pods are unaffected, so plan it in a window the workload accepts.
The systemd manual says restart stops and then starts a unit, and starts it if it is not running. The kubeadm pages use systemctl restart kubelet. Replace the unit with the one that the checks showed as stopped:
systemctl restart kubelet
systemctl is-active kubelet
To back out: a restart has no undo. The saved status and journal files record the state before it. If the unit does not return to active, read journalctl -u kubelet -n 100 --no-pager again and do not repeat the restart in a loop.
The machine is off, hung or isolated
First save the node's state. Run this from a machine that can reach the API server, and copy or photograph the console screen as well:
kubectl describe node k8s-node2 > /root/k8s-node2-2026-11-22.txt
Risk: powering on or resetting a node restarts every pod scheduled on it, and resetting a node that may still be writing data can lose that data.
Then start the machine or reach it through the hypervisor or provider console, and run the checks again. A power-on or reset cannot be undone; the saved output and the console record show the state before it. The control plane handles pods on its own schedule: with the default 5 minutes (300 seconds) between marking the node Unknown and the first eviction request, a node that returns within that time comes back before any eviction request is made for it.
The certificate or the API path
The kubeadm troubleshooting page covers a failed kubelet client certificate rotation with a multi-step repair that includes a new kubelet.conf. For an unreachable API server address, check routing and firewalls between the node and the API server, and "Kubernetes Networking Faults: CNI, Services, CoreDNS and How to Trace a Packet" for the network side.
Pressure conditions
If DiskPressure, MemoryPressure or PIDPressure is True, the node is short of that resource. The node status page defines each as True if pressure exists on disk size, memory or processes. That resource has to be freed first, as a separate task; no commands are given for it here.
Example (illustrative): k8s-node2 shows NotReady. kubectl describe node k8s-node2 shows Ready Unknown with NodeStatusUnknown and Kubelet stopped posting node status., the taint node.kubernetes.io/unreachable:NoExecute, and a Lease RenewTime earlier than the conditions' LastTransitionTime. SSH works. systemctl is-active kubelet reports the unit is not active, and the journal shows the kubelet exited. The engineer saves the status and journal, restarts the kubelet and checks again. If the node went Unknown at 10:00, the first eviction request would come at 10:05, since 10:00 plus 5 minutes is 10:05.
Confirm the fix
kubectl get nodesshows the nodeReady.kubectl describe nodeshows Ready True with reasonKubeletReady, and the LeaseRenewTimeadvancing.- The
Taints:line no longer listsnode.kubernetes.io/unreachable. systemctl is-active kubeletreports active, and no newx509errors appear in the kubelet journal.
What to watch afterwards
- Lease renewal time per node, so that a stalled heartbeat shows before the grace period passes.
- Kubelet and runtime service state, and restarts of either.
- Disk, memory and process pressure conditions on each node.
- The kubelet client certificate rotation on nodes built with kubeadm.
Read next
Related articles you may want next.
- TroubleshootingKubernetes Networking Faults: CNI, Services, CoreDNS and How to Trace a PacketSeparate DNS from Services, Services from the pod network, then capture packets at each hop to prove where a Kubernetes connection stops.Read article
- Troubleshootingkubeadm certificate has expired: "x509: certificate has expired or is not yet valid" and "Unauthorized" on a kubeadm clusterkubeadm cluster: certificate expired, kubectl says Unauthorized? Check expiry, back up /etc/kubernetes, renew and restart safely.Read article
- TroubleshootingLinux clock not synchronised: "System clock synchronized: no" with chrony and timedatectltimedatectl says System clock synchronized: no? Read chronyc tracking and sources, check UDP 123 and DNS, and know when chrony will step the clock.Read article
Risks and limits
- These commands have not been run or lab-tested.
- The Kubernetes pages give two default values for the unreachable timeout, 50 seconds and 40 seconds, so the real value depends on how your control plane is configured.
- The condition text comes from Kubernetes source code at release tag v1.34.0, where the node lifecycle controller sets it as a fixed string. The text can differ in other releases.
- The crictl commands need a Linux node with a CRI runtime. Managed Kubernetes services may not give you node access.
Sources
- Nodes — kubernetes.io, last modified 31 October 2026, accessed 22 November 2026
- Node Status — kubernetes.io, last modified 22 October 2025, accessed 22 November 2026
- Troubleshooting Clusters — kubernetes.io, last modified 4 July 2025, accessed 22 November 2026
- Debugging Kubernetes Nodes With Kubectl — kubernetes.io, last modified 27 June 2024, accessed 22 November 2026
- Debugging Kubernetes nodes with crictl — kubernetes.io, last modified 3 March 2025, accessed 22 November 2026
- Certificate Management with kubeadm — kubernetes.io, last modified 31 October 2026, accessed 22 November 2026
- Troubleshooting kubeadm — kubernetes.io, last modified 9 September 2026, accessed 22 November 2026
- systemctl — systemd manual page, no date shown (page version systemd 262), accessed 22 November 2026
- journalctl — systemd manual page, no date shown (page version systemd 262), accessed 22 November 2026
- node_lifecycle_controller.go — Kubernetes source code, release tag v1.34.0, no date shown, accessed 22 November 2026
This article was originally published on .