Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Kubernetes node NotReady: "Kubelet stopped posting node status"

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A node that shows NotReady, with conditions that read Kubelet stopped posting node status., has stopped sending status and heartbeats to the control plane. The node controller writes that message. It says contact was lost, not why. Check on the node whether the kubelet and the container runtime are running, whether the node can reach the API server, and whether its kubelet client certificate is valid.

What the symptom means

A node reports in two ways. The kubelet updates the node's .status, and it renews a Lease object in the kube-node-lease namespace. The node status page says the kubelet renews that Lease every 10 seconds by default, independently of status updates.

The Ready condition has three values. The node status page says it is True if the node is healthy, False if it is not healthy and not accepting pods, and Unknown "if the node controller has not heard from the node in the last node-monitor-grace-period (default is 50 seconds)". The same page's heartbeat section describes a 40 second default timeout for unreachable nodes, so read the value your cluster actually uses in the kube-controller-manager settings.

The text comes from the node lifecycle controller's source code (release tag v1.34.0), where a condition that has not been updated is set to Unknown with the reason NodeStatusUnknown and the message Kubelet stopped posting node status. The troubleshooting page shows the same text on the Ready, MemoryPressure, DiskPressure and PIDPressure conditions of a node that is down.

After the grace period the control plane adds a node.kubernetes.io/unreachable taint for Unknown, or node.kubernetes.io/not-ready for False. The nodes page says that by default the node controller waits 5 minutes between marking the node Unknown and submitting the first eviction request for its pods.

Likely causes

Cause What you see How to confirm
The kubelet has stopped or crashed Node Unknown; the machine still answers SSH systemctl is-active kubelet, then the journal
The container runtime is down The kubelet runs but cannot start or report containers crictl ps returns an error or does not answer on the node
The machine is off, hung or cut off from the network Node Unknown; no SSH, no ping Hypervisor or console view; network path to the API server
The kubelet's client certificate expired or failed to rotate The kubelet runs, yet the control plane rejects it x509: certificate has expired or is not yet valid in kube-apiserver logs
The node is out of disk, memory or process IDs Conditions DiskPressure, MemoryPressure or PIDPressure True kubectl describe node conditions
The node cannot reach the API server The kubelet journal shows connection errors to the API address Test from the node to the API server address and port

Prove the cause before changing anything

All of these are read-only.

1. Read the node's conditions, taints and lease.

kubectl get nodes
kubectl describe node k8s-node2

The troubleshooting page shows a down node with Conditions that carry NodeStatusUnknown and Kubelet stopped posting node status., a Taints line of node.kubernetes.io/unreachable:NoExecute, and a Lease whose RenewTime is earlier than the conditions' LastTransitionTime. A good result on a healthy node is Ready True. The node status page's healthy example shows reason KubeletReady and message kubelet is posting ready status.

2. Check the kubelet service on the node, if you can still log in.

systemctl is-active kubelet
systemctl status kubelet

The systemd manual says is-active returns an exit code 0 if a unit is active, and prints the current state. status shows runtime status followed by recent journal lines.

3. Read the kubelet's recent log.

journalctl -u kubelet -n 100 --no-pager

The journalctl manual describes -u as showing messages for a unit and -n as limiting the number of events shown. Look for repeated errors about the API server address, certificates or the runtime. The debugging page for clusters says that on systemd-based systems you may need journalctl instead of log files.

4. Check the container runtime.

crictl ps

The crictl page lists running containers with crictl ps. If it returns an error or does not answer, the runtime is the next thing to check; the page itself does not say so.

5. If you cannot log in, use the hypervisor or console. The node debugging page says kubectl debug node does not work when the node is down (disconnected from the network, or the kubelet dies and will not restart), so it is not a way in for this fault.

6. Check the kubelet certificate. If the log or the kube-apiserver log shows x509: certificate has expired or is not yet valid, the case is covered in our guide to expired kubeadm certificates and by the kubeadm troubleshooting page. Note that kubeadm certs check-expiration does not list kubelet.conf.

Fix

Fix the cause found. The kubelet's certificate case follows the kubeadm troubleshooting steps named in check 6.

The kubelet or the runtime service has stopped

First save the status of the unit you will restart and its recent log. The unit is kubelet unless the checks showed that the container runtime's unit has stopped; write that unit's name in place of <unit>:

systemctl status <unit> > /root/<unit>-status-2026-11-22.txt
journalctl -u <unit> -n 200 --no-pager > /root/<unit>-journal-2026-11-22.txt

Disruptive step: the restart command below restarts the kubelet on this node, which stops the kubelet and starts it again. If you restart the container runtime's unit instead, its containers can be interrupted. Kubernetes documentation gives no guarantee that running pods are unaffected, so plan it in a window the workload accepts.

The systemd manual says restart stops and then starts a unit, and starts it if it is not running. The kubeadm pages use systemctl restart kubelet. Replace the unit with the one that the checks showed as stopped:

systemctl restart kubelet
systemctl is-active kubelet

To back out: a restart has no undo. The saved status and journal files record the state before it. If the unit does not return to active, read journalctl -u kubelet -n 100 --no-pager again and do not repeat the restart in a loop.

The machine is off, hung or isolated

First save the node's state. Run this from a machine that can reach the API server, and copy or photograph the console screen as well:

kubectl describe node k8s-node2 > /root/k8s-node2-2026-11-22.txt

Risk: powering on or resetting a node restarts every pod scheduled on it, and resetting a node that may still be writing data can lose that data.

Then start the machine or reach it through the hypervisor or provider console, and run the checks again. A power-on or reset cannot be undone; the saved output and the console record show the state before it. The control plane handles pods on its own schedule: with the default 5 minutes (300 seconds) between marking the node Unknown and the first eviction request, a node that returns within that time comes back before any eviction request is made for it.

The certificate or the API path

The kubeadm troubleshooting page covers a failed kubelet client certificate rotation with a multi-step repair that includes a new kubelet.conf. For an unreachable API server address, check routing and firewalls between the node and the API server, and "Kubernetes Networking Faults: CNI, Services, CoreDNS and How to Trace a Packet" for the network side.

Pressure conditions

If DiskPressure, MemoryPressure or PIDPressure is True, the node is short of that resource. The node status page defines each as True if pressure exists on disk size, memory or processes. That resource has to be freed first, as a separate task; no commands are given for it here.

Example (illustrative): k8s-node2 shows NotReady. kubectl describe node k8s-node2 shows Ready Unknown with NodeStatusUnknown and Kubelet stopped posting node status., the taint node.kubernetes.io/unreachable:NoExecute, and a Lease RenewTime earlier than the conditions' LastTransitionTime. SSH works. systemctl is-active kubelet reports the unit is not active, and the journal shows the kubelet exited. The engineer saves the status and journal, restarts the kubelet and checks again. If the node went Unknown at 10:00, the first eviction request would come at 10:05, since 10:00 plus 5 minutes is 10:05.

Confirm the fix

  1. kubectl get nodes shows the node Ready.
  2. kubectl describe node shows Ready True with reason KubeletReady, and the Lease RenewTime advancing.
  3. The Taints: line no longer lists node.kubernetes.io/unreachable.
  4. systemctl is-active kubelet reports active, and no new x509 errors appear in the kubelet journal.

What to watch afterwards

  • Lease renewal time per node, so that a stalled heartbeat shows before the grace period passes.
  • Kubelet and runtime service state, and restarts of either.
  • Disk, memory and process pressure conditions on each node.
  • The kubelet client certificate rotation on nodes built with kubeadm.

Risks and limits

  • These commands have not been run or lab-tested.
  • The Kubernetes pages give two default values for the unreachable timeout, 50 seconds and 40 seconds, so the real value depends on how your control plane is configured.
  • The condition text comes from Kubernetes source code at release tag v1.34.0, where the node lifecycle controller sets it as a fixed string. The text can differ in other releases.
  • The crictl commands need a Linux node with a CRI runtime. Managed Kubernetes services may not give you node access.

Sources