Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Kubernetes pod Evicted: "The node was low on resource: ephemeral-storage" and DiskPressure

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A pod that stops with status Evicted and the message "The node was low on resource: ephemeral-storage." was failed by the kubelet on its node because a node filesystem fell below an eviction threshold. A pod that used more local storage than its own limit allows is evicted with a different message, for example "Container ... exceeded its local ephemeral storage limit". The wording of the message tells you which case you have. Find out what filled the disk before you change any threshold.

What the symptom means

The kubelet watches disk space and inodes on each node. In the Kubernetes documentation's words, when these "reach specific consumption levels, the kubelet can proactively fail one or more pods on the node". It sets the phase of the chosen pods to Failed. If a Deployment or StatefulSet owns the pod, its controller creates a replacement.

Three filesystem names appear in the documentation. nodefs is the node's main filesystem, which holds /var/lib/kubelet, log storage and emptyDir volumes. imagefs is an optional filesystem for container images. containerfs is an optional filesystem for writable container layers. The page notes that these names "do not always mean three different mount points".

Unless changed, the Linux defaults for the disk signals in hard eviction are nodefs.available<10%, imagefs.available<15%, nodefs.inodesFree<5% and imagefs.inodesFree<5%. A hard threshold has no grace period. The kubelet first tries to reclaim node-level space by removing dead containers and unused images, and evicts pods only if that is not enough.

These message texts come from the kubelet's source code at release tag v1.37.0, not from a documentation page. The source builds the message from the format strings "The node was low on resource: %v. " and "Threshold quantity: %v, available: %v. ". Each %v is replaced at run time: the first by the resource name, the others by the threshold and the space available. The resource name is ephemeral-storage for the nodefs.available, imagefs.available and containerfs.available signals, and inodes for the inode signals.

Separately, a pod can be evicted for breaking its own limit. The source uses these format strings in that case: "Container %s exceeded its local ephemeral storage limit %q. ", "Pod ephemeral local storage usage exceeds the total limit of containers %s. " and "Usage of EmptyDir volume %q exceeds the limit %q. ". The container or volume name and the limit are filled in at run time.

Likely causes

Cause What you see How to confirm
Node main filesystem below its threshold "The node was low on resource: ephemeral-storage." plus "Threshold quantity"; node condition DiskPressure is True df -h /var/lib/kubelet /var/log on the node shows available space under the threshold
Separate image filesystem below its threshold The same message, on a node with a dedicated image or container filesystem df -h on the container runtime's storage directory
Inodes exhausted "The node was low on resource: inodes." df -i shows few free inodes
One container passed its limit Container app exceeded its local ephemeral storage limit "4Gi". resources.limits.ephemeral-storage in the pod spec
Pod total passed the sum of its container limits "Pod ephemeral local storage usage exceeds the total limit of containers" and a size Sum the container limits; emptyDir use counts toward the total
emptyDir passed its sizeLimit Usage of EmptyDir volume "cache" exceeds the limit "500Mi". emptyDir.sizeLimit in the pod spec

Prove the cause before changing anything

All of these only read.

  1. Read the pod's status and message. Replace the names with yours:
kubectl get pod web-6d8f-x7k2p -n shop -o yaml

Read status.phase, status.reason and status.message. The kubelet source sets the reason to "Evicted". The message text matches one row of the table above.

  1. Read the node's conditions:
kubectl describe node k8s-node1

A DiskPressure condition that is True means a disk threshold is met. The taints page says the kubelet detects disk pressure from imagefs.available, imagefs.inodesFree, nodefs.available and nodefs.inodesFree.

  1. On the node, measure space and inodes. The ephemeral storage page names /var/log and /var/lib/kubelet as the kubelet's default directories:
df -h /var/lib/kubelet /var/log
df -i /var/lib/kubelet
du -xh --max-depth=1 /var/lib/kubelet

The df manual describes -h as human-readable sizes and -i as inode information. The du manual describes --max-depth=N and -x (stay on one filesystem). Compare the available figure with the threshold. A good result is available space above the threshold. A bad result is available space below it, or free inodes near zero.

  1. Read the limits the workload declares:
kubectl get deployment web -n shop -o yaml

Look for ephemeral-storage under resources.requests and resources.limits, and for sizeLimit under each emptyDir.

Example (illustrative): k8s-node1 has a 100 GiB main filesystem. The default nodefs.available<10% means eviction starts when less than 10 GiB is free (10% of 100 GiB). df -h shows 93 GiB used and 7 GiB available, so the threshold is met and kubectl describe node k8s-node1 shows DiskPressure True. du shows 52 GiB under the emptyDir of a report pod that has no ephemeral-storage limit. The fix is a limit on that pod, not a lower threshold.

Fix

Fix whatever is writing the data first. Lowering a threshold only delays the next eviction.

Set ephemeral-storage requests, limits and an emptyDir size

The documentation shows the fields resources.requests.ephemeral-storage, resources.limits.ephemeral-storage and emptyDir.sizeLimit. Watch the suffix case: it states that a request of 400m "is a request for 0.4 bytes", so write 400Mi or 400M. The limit for a pod is the sum of its container limits, and emptyDir use counts toward it. With one container limited to 10Gi and an emptyDir sizeLimit of 8Gi, the pod's total is 10 GiB, so the volume can use 8 of those 10 GiB.

First save the current manifest:

cp -p shop-web.yaml shop-web.yaml.2026-11-22

Disruptive step: the next change alters the pod template, and the Deployment documentation says a rollout is triggered when .spec.template changes, so the pods restart. A limit set too low evicts the pod as soon as it is exceeded.

Edit shop-web.yaml to add the fields, check it on the server without saving it, then apply it:

kubectl apply --dry-run=server -f shop-web.yaml
kubectl apply -f shop-web.yaml

To undo it: put the saved copy back under its original name, check it, then apply it:

cp -p shop-web.yaml.2026-11-22 shop-web.yaml
kubectl apply --dry-run=server -f shop-web.yaml
kubectl apply -f shop-web.yaml

Give the node more room

If legitimate use outgrew the disk, the remedy is more capacity on the filesystem that df showed, added through your platform's own tools. The ephemeral storage page lists the supported layouts: one filesystem, a separate runtime filesystem, or a split image filesystem. It warns that extra filesystems mounted under /var/lib/kubelet, /var/log or the runtime directory outside those layouts may leave the kubelet unable to report ephemeral storage correctly.

Do not run docker system prune or similar tools on a node. The garbage collection page says "You should avoid using external garbage collection tools, as these can break the kubelet behavior and remove containers that should exist." The kubelet already collects unused images every five minutes and unused containers every minute.

Stop repeated evictions with a minimum reclaim

The documentation says that when an eviction frees only a little, "the kubelet repeatedly hitting the configured eviction thresholds" can trigger multiple evictions, and evictionMinimumReclaim sets how much to reclaim; the default is 0. On a node built with kubeadm the page names /var/lib/kubelet/config.yaml as the kubelet configuration file. Other installers may use another path.

First save the current file:

cp -p /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.2026-11-22

Disruptive step: the next change ends with systemctl restart kubelet, which restarts the kubelet on this one node. The kubeadm page says to do such changes one node at a time. Do not edit evictionHard here. The documentation warns that changing one evictionHard value sets the others to zero unless you list them all or set MergeDefaultEvictionSettings to true.

Add this to the file, then restart the kubelet. If the file already has an evictionMinimumReclaim key, add the nodefs.available line under it instead of adding a second key:

evictionMinimumReclaim:
  nodefs.available: "500Mi"
systemctl restart kubelet

To back out: restore the saved file under its original name, then restart:

cp -p /var/lib/kubelet/config.yaml.2026-11-22 /var/lib/kubelet/config.yaml
systemctl restart kubelet

Confirm the fix

  1. kubectl describe node k8s-node1 shows DiskPressure False.
  2. df -h /var/lib/kubelet shows available space above the threshold with room to spare.
  3. Over the next working day no new pod in the namespace reports status.reason Evicted when you read it with kubectl get pod NAME -n shop -o yaml.
  4. The workload's kubectl get deployment web -n shop -o yaml shows the limits you set.

What to watch afterwards

  • Available space on /var/lib/kubelet, /var/log and the runtime directory, with an alert well above the eviction threshold.
  • Free inodes, as well as free space.
  • New "Evicted" pods with a "Container ... exceeded" or "Usage of EmptyDir volume" message. They point at one workload.
  • After a kubeadm upgrade, the file /var/lib/kubelet/config.yaml. The kubeadm page says an upgrade can overwrite it with the cluster-wide ConfigMap.

Risks and limits

  • These commands have not been run or lab-tested.
  • The documentation says that if the kubelet is not measuring local ephemeral storage, "a Pod that exceeds its local storage limit will not be evicted for breaching local storage resource limits."
  • Message wording comes from the kubelet's source code at release tag v1.37.0 and can change between releases. Match your cluster's text against the source of its own version.
  • Eviction uses no grace period for hard thresholds, and the kubelet does not respect a PodDisruptionBudget during node-pressure eviction.
  • Windows nodes use different default thresholds and are not covered.

Sources