Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Kubernetes FailedScheduling: "0/3 nodes are available: 3 Insufficient cpu" (or memory)

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A pod that stays Pending with the event FailedScheduling and a message such as 0/3 nodes are available: 3 Insufficient cpu. could not be placed on any node. The scheduler counts what pods have asked for, not what they use. Read the reason counts in the message, compare the pod's requests with each node's allocatable amount, then lower the request, free capacity or add a node.

What the symptom means

The Kubernetes resource page says that when the scheduler cannot find a node where a pod fits, the pod remains unscheduled until a place can be found. An event is produced each time. Its example reads 0/42 nodes available: insufficient cpu with the reason FailedScheduling from default-scheduler.

The longer message text comes from the scheduler's source code (release tag v1.34.0), not from a documentation page. The source builds it from the format 0/%v nodes are available followed by a colon, where %v is the number of nodes in the cluster. After the colon it adds one entry for each reason, written as a count then the reason (for example 3 Insufficient cpu), sorted and separated by commas, and ending in a full stop. The parts that vary are the node count, each count and each reason. The source also adds a message from a pre-filter plugin in place of the counts when one rejects the pod, and appends the post-filter (preemption) message when there is one, so a real event can carry more text after the counts. The reasons for resource checks are Insufficient cpu, Insufficient memory, Insufficient ephemeral-storage and Too many pods. Other filters add node(s) had taints that the pod didn't tolerate, node(s) didn't match Pod's node affinity/selector and node(s) were unschedulable. Each count is a number of nodes, so 1 node(s) had taints that the pod didn't tolerate, 2 Insufficient cpu names the cause node by node. A node can fail for more than one resource, so the counts can add up to more than N.

The check uses requests. The resource page says the scheduler "ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node", and that it refuses the pod even when actual use is very low. In the source, a node fails the CPU check when the pod's request is greater than the node's allocatable CPU minus the requests already on it.

Capacity and allocatable are different. The resource page says the amount available to pods is less than node capacity because system daemons use a portion, and .status.allocatable is the figure the scheduler uses.

Likely causes

Cause What you see How to confirm
Requests already on each node leave too little CPU or memory N Insufficient cpu or N Insufficient memory for the nodes concerned kubectl describe node shows Allocatable and Allocated resources; free = Allocatable minus requests
The request is larger than any node's allocatable The same message on every node, even on an empty cluster Compare the pod's request with the largest Allocatable value
The request has a wrong unit A request far bigger or smaller than intended kubectl get pod -o yaml shows values; 100m is 0.1 CPU and 100 is 100 CPUs
Nodes carry taints the pod does not tolerate node(s) had taints that the pod didn't tolerate Taints: in kubectl describe node, and tolerations in the pod
A nodeSelector or affinity matches no node node(s) didn't match Pod's node affinity/selector Compare the pod's selector with the node labels
Nodes are cordoned node(s) were unschedulable The Unschedulable: line in kubectl describe node shows true for a cordoned node; the Troubleshooting Clusters page shows that line, with false, on an uncordoned node
The node is at its pod limit Too many pods pods: under Allocatable compared with the pods running on the node

Prove the cause before changing anything

All of these are read-only.

1. Read the exact message and its counts.

kubectl describe pod <pod-name> -n shop

Look at the Events section. Each count in the message is the number of nodes that failed for that reason, out of the N in 0/N.

2. Read each node's capacity, allocatable amount and allocated requests.

kubectl describe node <node-name>

Under Capacity and Allocatable are cpu, memory and pods. Under Allocated resources are the summed requests. The resource page works one example: with Allocatable CPU of 1800m and 680m requested, a pod asking for more than 1.120 CPUs will not fit, because 1800m − 680m = 1120m.

3. Read the pod's requests.

kubectl get pod <pod-name> -n shop -o yaml

Find resources.requests for every container. The pod's request is the sum over its containers. Check CPU units: the resource page says 0.1 is equivalent to 100m. Check memory units: the page warns that 400m of memory is 0.4 bytes, and that 400 mebibytes is 400Mi.

4. Read taints and cordon state. The same kubectl describe node output shows Taints: and Unschedulable:. The Taints and Tolerations page gives NoSchedule as: no new pods are scheduled on the tainted node unless they have a matching toleration, and pods already running there are not evicted.

5. Read labels if a selector is set. The Labels: lines in kubectl describe node <node-name> list each node's labels. A good result is a node whose labels satisfy every selector or affinity term in the pod. A bad result is no such node.

Fix

Fix only the cause the message and the node figures showed.

The request is too large for the free space

First save the Deployment, which holds the present requests:

kubectl get deployment report -n shop -o yaml > /root/report-deployment-2026-11-22.yaml

Risk: the request change below edits the pod template of the report Deployment. That starts a rollout, and a smaller request means the scheduler may place more pods on a node than before.

The reference lists --requests for kubectl set resources and server as a --dry-run value. Run the first command to check, then the second to apply:

kubectl set resources deployment report -n shop -c=report --requests=cpu=250m --dry-run=server
kubectl set resources deployment report -n shop -c=report --requests=cpu=250m

To undo it, set the request back to the value in the saved file (here 500m) and read the rollout:

kubectl set resources deployment report -n shop -c=report --requests=cpu=500m
kubectl rollout status deployment/report -n shop

The scheduler places pods by requests, so a request set below real use lets a node be packed beyond what its pods need. The resource page says that when a container exceeds its memory request and the node becomes short of memory overall, it is likely that the pod will be evicted. Check real usage before a request is lowered.

Capacity is full

The resource page lists three remedies: add more nodes to the cluster, terminate unneeded pods to make room, and check that the pod is not larger than all the nodes. It gives the example that if all nodes have cpu: 1, a pod requesting cpu: 1.1 will never be scheduled. Adding a node is a change to the cluster's infrastructure and follows your own provisioning process.

A taint blocks the pod

The Taints page describes removing a taint with a trailing minus. First read the exact taint on the node and save it:

kubectl describe node k8s-node3 > /root/k8s-node3-2026-11-22.txt

Risk: removing the taint below lets every pod on the cluster that does not tolerate it be scheduled onto k8s-node3, which defeats the purpose if the taint reserves the node.

Copy the taint exactly as the saved Taints: line shows it. For a taint dedicated=batch:NoSchedule:

kubectl taint nodes k8s-node3 dedicated=batch:NoSchedule-

To undo it, add the same taint back and read the node:

kubectl taint nodes k8s-node3 dedicated=batch:NoSchedule
kubectl describe node k8s-node3

A taint that is there on purpose is better answered with a toleration in the pod spec, described on the same page.

A selector or a cordon

A selector that matches no node is corrected in the pod spec by the workload's owner, or the missing label is added to a node by a cluster administrator. For a cordoned node, the node maintenance page describes kubectl uncordon <node-name> as the way to tell Kubernetes it can resume scheduling new pods onto the node. That suits only a node whose maintenance has finished, and it is a decision for whoever cordoned the node.

Example (illustrative): deployment report in namespace shop has a pod asking for cpu: 500m. The nodes k8s-node1, k8s-node2 and k8s-node3 each have Allocatable CPU of 1800m. k8s-node1 has 1500m requested, leaving 300m (1800 − 1500). k8s-node2 has 1400m requested, leaving 400m. k8s-node3 carries the taint dedicated=batch:NoSchedule. The event reads 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 Insufficient cpu. The taint is deliberate, so the engineer saves the Deployment and lowers the request to 250m. That is less than 300m and 400m, so the pod fits on either of the first two nodes.

Confirm the fix

  1. kubectl get pods -n shop shows the pod leaving Pending and reaching Running.
  2. kubectl describe pod shows a Scheduled event and no new FailedScheduling event.
  3. kubectl describe node on the chosen node shows allocated requests still below Allocatable.

What to watch afterwards

  • Allocated requests against Allocatable on each node, per CPU and per memory, as a percentage.
  • Pending pods after each release, since larger requests arrive with new versions.
  • Taints and cordons left in place after maintenance.
  • Pod counts against the pods limit on nodes that run many small pods.

Risks and limits

  • These commands have not been run or lab-tested.
  • The wording of each reason comes from Kubernetes source code at release tag v1.34.0. Other releases can word reasons differently, so match on meaning.
  • The resource page's example message is shorter than the messages the source builds, so expect the longer form with counts.
  • Scheduler preemption, priority and autoscaler behaviour can change which pods are pending and are not covered here.

Sources