About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
A pod that stops with status Evicted and the message "The node was low on resource: ephemeral-storage." was failed by the kubelet on its node because a node filesystem fell below an eviction threshold. A pod that used more local storage than its own limit allows is evicted with a different message, for example "Container ... exceeded its local ephemeral storage limit". The wording of the message tells you which case you have. Find out what filled the disk before you change any threshold.
What the symptom means
The kubelet watches disk space and inodes on each node. In the Kubernetes documentation's words, when these "reach specific consumption levels, the kubelet can proactively fail one or more pods on the node". It sets the phase of the chosen pods to Failed. If a Deployment or StatefulSet owns the pod, its controller creates a replacement.
Three filesystem names appear in the documentation. nodefs is the node's main filesystem, which holds /var/lib/kubelet, log storage and emptyDir volumes. imagefs is an optional filesystem for container images. containerfs is an optional filesystem for writable container layers. The page notes that these names "do not always mean three different mount points".
Unless changed, the Linux defaults for the disk signals in hard eviction are nodefs.available<10%, imagefs.available<15%, nodefs.inodesFree<5% and imagefs.inodesFree<5%. A hard threshold has no grace period. The kubelet first tries to reclaim node-level space by removing dead containers and unused images, and evicts pods only if that is not enough.
These message texts come from the kubelet's source code at release tag v1.37.0, not from a documentation page. The source builds the message from the format strings "The node was low on resource: %v. " and "Threshold quantity: %v, available: %v. ". Each %v is replaced at run time: the first by the resource name, the others by the threshold and the space available. The resource name is ephemeral-storage for the nodefs.available, imagefs.available and containerfs.available signals, and inodes for the inode signals.
Separately, a pod can be evicted for breaking its own limit. The source uses these format strings in that case: "Container %s exceeded its local ephemeral storage limit %q. ", "Pod ephemeral local storage usage exceeds the total limit of containers %s. " and "Usage of EmptyDir volume %q exceeds the limit %q. ". The container or volume name and the limit are filled in at run time.
Likely causes
| Cause | What you see | How to confirm |
|---|---|---|
| Node main filesystem below its threshold | "The node was low on resource: ephemeral-storage." plus "Threshold quantity"; node condition DiskPressure is True | df -h /var/lib/kubelet /var/log on the node shows available space under the threshold |
| Separate image filesystem below its threshold | The same message, on a node with a dedicated image or container filesystem | df -h on the container runtime's storage directory |
| Inodes exhausted | "The node was low on resource: inodes." | df -i shows few free inodes |
| One container passed its limit | Container app exceeded its local ephemeral storage limit "4Gi". |
resources.limits.ephemeral-storage in the pod spec |
| Pod total passed the sum of its container limits | "Pod ephemeral local storage usage exceeds the total limit of containers" and a size | Sum the container limits; emptyDir use counts toward the total |
emptyDir passed its sizeLimit |
Usage of EmptyDir volume "cache" exceeds the limit "500Mi". |
emptyDir.sizeLimit in the pod spec |
Prove the cause before changing anything
All of these only read.
- Read the pod's status and message. Replace the names with yours:
kubectl get pod web-6d8f-x7k2p -n shop -o yaml
Read status.phase, status.reason and status.message. The kubelet source sets the reason to "Evicted". The message text matches one row of the table above.
- Read the node's conditions:
kubectl describe node k8s-node1
A DiskPressure condition that is True means a disk threshold is met. The taints page says the kubelet detects disk pressure from imagefs.available, imagefs.inodesFree, nodefs.available and nodefs.inodesFree.
- On the node, measure space and inodes. The ephemeral storage page names
/var/logand/var/lib/kubeletas the kubelet's default directories:
df -h /var/lib/kubelet /var/log
df -i /var/lib/kubelet
du -xh --max-depth=1 /var/lib/kubelet
The df manual describes -h as human-readable sizes and -i as inode information. The du manual describes --max-depth=N and -x (stay on one filesystem). Compare the available figure with the threshold. A good result is available space above the threshold. A bad result is available space below it, or free inodes near zero.
- Read the limits the workload declares:
kubectl get deployment web -n shop -o yaml
Look for ephemeral-storage under resources.requests and resources.limits, and for sizeLimit under each emptyDir.
Example (illustrative): k8s-node1 has a 100 GiB main filesystem. The default nodefs.available<10% means eviction starts when less than 10 GiB is free (10% of 100 GiB). df -h shows 93 GiB used and 7 GiB available, so the threshold is met and kubectl describe node k8s-node1 shows DiskPressure True. du shows 52 GiB under the emptyDir of a report pod that has no ephemeral-storage limit. The fix is a limit on that pod, not a lower threshold.
Fix
Fix whatever is writing the data first. Lowering a threshold only delays the next eviction.
Set ephemeral-storage requests, limits and an emptyDir size
The documentation shows the fields resources.requests.ephemeral-storage, resources.limits.ephemeral-storage and emptyDir.sizeLimit. Watch the suffix case: it states that a request of 400m "is a request for 0.4 bytes", so write 400Mi or 400M. The limit for a pod is the sum of its container limits, and emptyDir use counts toward it. With one container limited to 10Gi and an emptyDir sizeLimit of 8Gi, the pod's total is 10 GiB, so the volume can use 8 of those 10 GiB.
First save the current manifest:
cp -p shop-web.yaml shop-web.yaml.2026-11-22
Disruptive step: the next change alters the pod template, and the Deployment documentation says a rollout is triggered when .spec.template changes, so the pods restart. A limit set too low evicts the pod as soon as it is exceeded.
Edit shop-web.yaml to add the fields, check it on the server without saving it, then apply it:
kubectl apply --dry-run=server -f shop-web.yaml
kubectl apply -f shop-web.yaml
To undo it: put the saved copy back under its original name, check it, then apply it:
cp -p shop-web.yaml.2026-11-22 shop-web.yaml
kubectl apply --dry-run=server -f shop-web.yaml
kubectl apply -f shop-web.yaml
Give the node more room
If legitimate use outgrew the disk, the remedy is more capacity on the filesystem that df showed, added through your platform's own tools. The ephemeral storage page lists the supported layouts: one filesystem, a separate runtime filesystem, or a split image filesystem. It warns that extra filesystems mounted under /var/lib/kubelet, /var/log or the runtime directory outside those layouts may leave the kubelet unable to report ephemeral storage correctly.
Do not run docker system prune or similar tools on a node. The garbage collection page says "You should avoid using external garbage collection tools, as these can break the kubelet behavior and remove containers that should exist." The kubelet already collects unused images every five minutes and unused containers every minute.
Stop repeated evictions with a minimum reclaim
The documentation says that when an eviction frees only a little, "the kubelet repeatedly hitting the configured eviction thresholds" can trigger multiple evictions, and evictionMinimumReclaim sets how much to reclaim; the default is 0. On a node built with kubeadm the page names /var/lib/kubelet/config.yaml as the kubelet configuration file. Other installers may use another path.
First save the current file:
cp -p /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.2026-11-22
Disruptive step: the next change ends with systemctl restart kubelet, which restarts the kubelet on this one node. The kubeadm page says to do such changes one node at a time. Do not edit evictionHard here. The documentation warns that changing one evictionHard value sets the others to zero unless you list them all or set MergeDefaultEvictionSettings to true.
Add this to the file, then restart the kubelet. If the file already has an evictionMinimumReclaim key, add the nodefs.available line under it instead of adding a second key:
evictionMinimumReclaim:
nodefs.available: "500Mi"
systemctl restart kubelet
To back out: restore the saved file under its original name, then restart:
cp -p /var/lib/kubelet/config.yaml.2026-11-22 /var/lib/kubelet/config.yaml
systemctl restart kubelet
Confirm the fix
kubectl describe node k8s-node1shows DiskPressure False.df -h /var/lib/kubeletshows available space above the threshold with room to spare.- Over the next working day no new pod in the namespace reports
status.reasonEvicted when you read it withkubectl get pod NAME -n shop -o yaml. - The workload's
kubectl get deployment web -n shop -o yamlshows the limits you set.
What to watch afterwards
- Available space on
/var/lib/kubelet,/var/logand the runtime directory, with an alert well above the eviction threshold. - Free inodes, as well as free space.
- New "Evicted" pods with a "Container ... exceeded" or "Usage of EmptyDir volume" message. They point at one workload.
- After a kubeadm upgrade, the file
/var/lib/kubelet/config.yaml. The kubeadm page says an upgrade can overwrite it with the cluster-wide ConfigMap.
Read next
Related articles you may want next.
- TroubleshootingDocker no space left on device: measure /var/lib/docker before you pruneDocker no space left on device: find what fills /var/lib/docker with docker system df, rotate logs, and prune images and cache without losing volumes.Read article
- TroubleshootingJournal logs filling the disk: how to limit /var/log/journal and vacuum it safely/var/log/journal is huge? Check journalctl --disk-usage, trim with --vacuum-size, and cap it with SystemMaxUse. What is lost, and what vacuum cannot touch.Read article
- TroubleshootingKubernetes FailedScheduling: "0/3 nodes are available: 3 Insufficient cpu" (or memory)Pod Pending with FailedScheduling and Insufficient cpu or memory? Read the reason counts, compare requests to allocatable, then fix the cause.Read article
Risks and limits
- These commands have not been run or lab-tested.
- The documentation says that if the kubelet is not measuring local ephemeral storage, "a Pod that exceeds its local storage limit will not be evicted for breaching local storage resource limits."
- Message wording comes from the kubelet's source code at release tag v1.37.0 and can change between releases. Match your cluster's text against the source of its own version.
- Eviction uses no grace period for hard thresholds, and the kubelet does not respect a PodDisruptionBudget during node-pressure eviction.
- Windows nodes use different default thresholds and are not covered.
Sources
- Node-pressure Eviction — Kubernetes documentation, last modified 31 October 2026, accessed 22 November 2026
- Local ephemeral storage — Kubernetes documentation, last modified 27 September 2026, accessed 22 November 2026
- Garbage Collection — Kubernetes documentation, last modified 31 October 2026, accessed 22 November 2026
- Set Kubelet Parameters Via A Configuration File — Kubernetes documentation, last modified 28 July 2026, accessed 22 November 2026
- Reconfiguring a kubeadm cluster — Kubernetes documentation, last modified 17 December 2024, accessed 22 November 2026
- Well-Known Labels, Annotations and Taints — Kubernetes documentation, last modified 27 July 2026, accessed 22 November 2026
- Deployments — Kubernetes documentation, last modified 7 July 2026, accessed 22 November 2026
- kubectl get, kubectl describe and kubectl apply — Kubernetes documentation, last modified 26 August 2026, accessed 22 November 2026
- kubelet eviction helpers.go, tag v1.37.0 — Kubernetes source code, no date shown, accessed 22 November 2026
- kubelet eviction_manager.go, tag v1.37.0 — Kubernetes source code, no date shown, accessed 22 November 2026
- df(1), du(1) and cp(1) — GNU coreutils 9.11 manual pages, April 2026, accessed 22 November 2026
- systemctl — systemd documentation, no date shown, accessed 22 November 2026
This article was originally published on .