CPU overcommit in hosted clouds: ready time, NUMA and how to size honestly

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Overcommit is normal, and some vCPU wait comes with it. It becomes a problem when wait lines up with a real symptom or climbs above your own baseline, so size from measured wait, not from a ratio. None of the VMware, Proxmox or Microsoft documents checked for this article gives a general vCPU-to-core ratio to aim for. They give measurements and warning levels, which is what this article uses.

The practical consequence: a slow VM with a high wait figure often needs fewer vCPUs or a quieter host, not more vCPUs.

Commands below were not run in a lab. VM IDs and host sizes are examples.

Recommendations and interpretations are editorial unless attributed; source research is not lab validation.

Why it matters

A vCPU is a share of scheduler time, not a reserved core. VMware's vSphere 8.0 best practices (Revision 20221223, so later 8.0 updates may not be reflected) say that in most environments ESXi allows "significant levels" of overcommitment without affecting VM performance, and that latency-sensitive workloads may suffer once a host is saturated. Proxmox VE calls it "perfectly safe" for the total of VM cores to exceed server cores, but refuses to start a single VM with more vCPUs than the host physically has.

The guest cannot explain the problem on its own. Broadcom's Guest SDK documentation says the individual reasons a vCPU made no progress are hidden from the guest to preserve isolation. The guest sees one combined figure at most. That matters when comparing hosting providers: the tenant holds half the evidence and the provider holds the rest.

Check what a ratio is counted against

The two big vendors define the term differently. VMware counts vCPUs against physical cores. Microsoft's Windows Server 2025 post counts virtual processors against logical processors.

Take a hypothetical host with 64 cores, 128 threads and 256 vCPUs allocated. That is 4:1 against cores and 2:1 against threads, for the same machine.

Threads are not cores. Broadcom states that an application on one logical processor of a busy core can expect slightly more than half the throughput it gets running alone on a non-hyperthreaded processor.

How it works

Every hypervisor keeps a queue of runnable vCPUs. The figure you need is how long a vCPU had work to do and could not get a processor.

ESXi: read ready time correctly

Broadcom's CPU panel reference defines %RDY as time the VM was ready to run but was not provided CPU. %MLMTD is time the VMkernel deliberately did not run it because of a limit, and that time is included in %RDY.

So subtract before blaming neighbours. Broadcom's Guest SDK page treats %RDY - %MLMTD as time lost to host CPU overutilisation and %MLMTD as time lost to configured limits. The CPU panel reference gives the identity 100% = %RUN + %RDY + %CSTP + %WAIT.

vSphere charts report ready as milliseconds accumulated over a sample period. Broadcom KB 387750 gives the conversion:

ready % = (summation in ms x 100) / interval in ms
per-vCPU ready % = ready % / number of vCPUs

Worked example, using invented figures: a 4-vCPU VM shows 30,000 ms on the Last Day chart.

30,000 x 100 / 300,000 = 10% for the VM
10% / 4 vCPUs = 2.5% per vCPU

Two mistakes follow from skipping this arithmetic. Reading the 10% as a per-vCPU value makes a healthy VM look bad, because the chart and esxtop both show the sum across vCPUs. Reading 2.5% as the whole story hides bursts. By arithmetic, a five-minute average of 2.5% per vCPU could be 37.5% during one 20-second interval and zero during the other fourteen.

In esxtop, expand a VM's group with the interactive e command to see each world, then read per-vCPU values.

Broadcom KB 438023 calls these bands "industry-accepted" for per-vCPU ready:

The same KB says sustained high ready points to host contention rather than an undersized VM. KB 387750 adds an important limit: ready "in itself is not an indicator for existing performance issues". Treat it as a tool for explaining a real symptom, not an alarm on its own.

Co-stop: too many vCPUs scheduled together

%CSTP applies to multi-vCPU VMs. KB 438023 treats below about 3% per vCPU as normal, and says higher sustained values usually mean more vCPUs than the VM can use. KB 1005362 says that above 3.00 the performance issues "may be caused by" the vCPU count, and suggests trying one fewer vCPU, with the VM powered off.

The two KBs differ on the basis. KB 438023 says per vCPU; KB 1005362 does not say. Because esxtop's VM row is a sum across vCPUs, 3.00 on the row of an 8-vCPU VM is only 0.375 per vCPU. Our reading: expand the VM with e, compare each world against 3, and treat a VM-row figure as a prompt to look.

The esxtop overview KB lists %RDY at 10 or more, says to "take care" between five and ten percent, and does not say whether that is per vCPU. Read per vCPU, those figures match the KB 438023 bands above, and we use the per-vCPU basis because KB 438023 states it. The same KB also lists %MLMTD at 1 or more and %SWPWT at 5 or more as attention levels. Swap wait is a memory problem presenting as slow CPU work.

Proxmox VE and KVM: there is no ready counter by name

Each vCPU is a QEMU thread that the Linux scheduler balances like any other. The kernel scheduler statistics expose /proc/<pid>/schedstat, with three fields: time on CPU in nanoseconds, time waiting on a runqueue in nanoseconds, and timeslices run.

Read it per thread, twice, and work out the rate. The per-thread files live under /proc/<pid>/task/, and comm gives each thread's name. Replace <pid> with the VM's QEMU process ID. This is read-only and was not tested for this article.

for t in /proc/<pid>/task/*; do
  echo "$(cat "$t/comm") $(cat "$t/schedstat")"
done

If a vCPU thread's second field rises by 2,000,000,000 over 20 seconds, it spent 10% of that window queued (2 seconds out of 20). This calculation is the author's method. Proxmox publishes no warning level for it, so the ESXi bands above are only a rough analogy. The process also has non-vCPU threads. QEMU names vCPU threads CPU 0/KVM, CPU 1/KVM and so on, and Proxmox VE turns that naming on, so compare only those lines.

Limits look like contention from inside the guest. cpulimit maps to CPUQuota and cpuunits to CPUWeight. In cgroup v2, rising nr_throttled and throttled_usec in cpu.stat show the quota is biting. This is the KVM equivalent of %MLMTD. cpuunits is a relative weight, default 100 (1024 on a host still using cgroup v1). In Proxmox's own example, a VM at 200 receives twice the CPU bandwidth of a VM at 100.

Pressure stall information gives a host-wide trend through /proc/pressure/cpu:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of time at least some tasks were stalled for CPU. The kernel states that CPU full is undefined at system level and is reported as zero for compatibility, so do not read it. Each cgroup has its own cpu.pressure.

Inside a Linux guest, /proc/stat reports steal and vmstat shows it as st. On KVM, the steal time interface counts time the vCPU did not run and excludes idle time. A steady zero is only useful once you have seen the counter move under known load.

Hyper-V: multiply two counters

Microsoft's bottleneck guide treats \Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time above 90% as an overloaded host. If all of a VM's virtual processors are above 90%, it says to confirm the host is not overloaded, find out whether the workload can use more virtual processors, and then assign more.

Windows Server 2025 adds a sharper measure. The Microsoft post of 13 March 2025 gives:

Contention delay time =
  CPU Contention Time Per Dispatch x Logical Processor Dispatches/sec
  [nanoseconds of contention delay per second]

The post warns that the per-dispatch figure on its own can mislead, because workloads dispatch at very different rates. CPU Wait Time Per Dispatch is the older combined figure, and does not say why the processor waited. This source is a Microsoft engineering blog, not a counter reference, and it publishes no threshold. Set a baseline on your own hosts.

The scheduler type changes what headroom means. The core scheduler has been the default since Windows Server 2019. It schedules SMT pairs together and does not share a core between VMs. Microsoft notes it "can potentially reduce performance": if only one virtual processor in a group can run, the other thread on that core sits idle. A count of idle logical processors can therefore overstate the room left. Microsoft also recommends even vCPU counts when SMT is enabled; our reading is that this gives the scheduler full pairs to fill.

SQL Server: confirm the queue from inside

sys.dm_os_wait_stats records signal_wait_time_ms, the gap between a thread being signalled and starting to run. sys.dm_os_schedulers shows runnable_tasks_count, tasks waiting for a scheduler.

SELECT scheduler_id, runnable_tasks_count, work_queue_count, pending_disk_io_count
FROM sys.dm_os_schedulers
WHERE status = 'VISIBLE ONLINE';

These show a queue, not its cause. Microsoft says prolonged SOS_SCHEDULER_YIELD waits most often point to query optimisation opportunities, so check query work before asking for more vCPUs.

NUMA: the second way to lose performance

A NUMA node is a group of cores with its local memory. Remote memory is slower to reach.

ESXi gives each VM a home node, constrains its vCPUs to that node, and can move the home node and migrate memory later. A VM with more vCPUs than one node has cores is a wide VM, split across several nodes, and may see higher average memory latency.

Because vCPUs stay on their home node, one node can be busy while the host average looks comfortable. That is the author's inference from the placement rule. Check per-node load before concluding a host has spare capacity.

Evidence on ESXi comes from the memory panel. NHN is the current home node, NRMEM is remote memory and N%L is the local percentage. The esxtop KB treats N%L at or below 80 as a performance warning.

Automatic vTopology needs hardware version 20 or later, with Cores per Socket left at "Assigned at power on". The 8.0 controls state that numa.vcpu.maxPerVirtualNode defaults to 8 and that cores per socket does not set vNUMA unless numa.vcpu.followcorespersocket is configured.

One conflict needs checking on your own build. KB 438023 says vNUMA is preserved with CPU Hot Add on hardware version 20. The 8.0 Virtual Topology page says a VM with CPU hotplug enabled enforces a single NUMA node by default. Run numactl --hardware in a Linux guest to see what the guest actually received.

On Proxmox VE the NUMA option defaults to off. The documentation recommends it when the host has more than one node and says to set sockets to the node count. numa[n] entries can bind guest nodes to host nodes.

numactl --hardware | grep available
numastat -p qemu

numastat -p shows per-node memory for matching processes, so a VM spread across nodes it should not use will be visible. Kernel automatic NUMA balancing has a cost from unmapping pages and trapping faults, and the kernel says to disable it if the workload is already bound to nodes.

On Hyper-V, vNUMA matches the host by default, and only NUMA-aware applications benefit from it. Microsoft also states that Virtual NUMA and dynamic memory cannot be used together: a VM with dynamic memory enabled effectively has one NUMA node.

What to choose and why

The method below is the author's recommendation, built from the vendor figures above.

Step 1: Capture demand. KB 438023 suggests a representative period of typically 24 hours, with guest CPU at or below 80% sustained. Add your own busiest periods, such as month-end, to that window.

Step 2: Name the cause before changing anything.

Step 3: Fit the VM to the node. KB 438023 advises splitting wide VMs evenly across the fewest nodes, avoiding odd vCPU counts that cross nodes, and keeping the count within the host's physical cores.

Step 4: Plan the host for failure. VMware's guide calls 80% a reasonable host ceiling and 90% a warning, and notes organisations vary. Apply it after a host loss. In a four-host cluster where one host fails, load rises by a factor of 4/3, so staying at 80% afterwards means running near 60% beforehand. This arithmetic ignores restart priorities, reservations and uneven VM sizes.

Step 5: Remeasure. Compare the same counters, over a comparable period, after the change.

Questions for a hosted cloud provider

• Is the ratio counted against cores or threads?
• Can you see per-vCPU wait for your VMs, or will the provider supply it on request?
• Are vCPUs reserved, weighted or limited? On ESXi, a guest with VMware Tools can read vm.cpu.reserved and vm.cpu.limit.
• What is the physical core count and memory per NUMA node, and is it the same across hosts?
• How much capacity is held back for host failure and maintenance?

From inside a VMware guest, vmware-toolbox-cmd stat raw text resources returns cumulative counters including vm.cpu.contention.cpu. Broadcom's formula is 100 x (change in contention) / (vCPUs x elapsed time). It combines several causes, including limits, so treat it as a reason to ask questions.

Risks and limits

• Do not add vCPUs in response to ready time alone. KB 438023 says extra vCPUs add scheduling overhead, and high ready signals contention.
• Resizing on ESXi is a change window. The rightsizing procedure in KB 438023 says changes to vCPU count, memory or topology need the VM powered down, even where CPU Hot Add stays enabled.
• Avoid CPU affinity as a fix. Broadcom warns that with manual affinity the NUMA scheduler may be unable to manage the VM. Proxmox states that affinity is not a security feature and does not cover peripheral I/O processes.
• numa_balancing is host-wide. Changing it affects every VM on that host, so do not change it to fix one guest without testing.
• The thresholds are guidance. Broadcom's bands are general, and a latency-sensitive workload may suffer below them. No source here gives a limit for Proxmox or Hyper-V wait figures.
• Not covered: memory overcommit, side-channel security and scheduler mitigations, licensing per core, and GPU or passthrough workloads.
• Version limits. The Guest SDK page is the 7.0 edition, the best practices guide is Revision 20221223, and KB 1005362 names no ESXi version. The Windows Server 2025 counters come from a Microsoft blog. Check your own build.

Sources

Opened 2026-11-10.

• Performance Best Practices for VMware vSphere 8.0
• Broadcom CPU panel (esxtop, vSphere 8.0)
• Broadcom memory panel (esxtop, vSphere 8.0)
• Broadcom KB 387750: Understanding the CPU ready values
• Broadcom KB 438023: Rightsizing virtual machines on ESXi 8.0
• Broadcom KB 308290: esxtop overview
• Broadcom KB 1005362: co-stop
• How ESXi NUMA scheduling works (8.0)
• Virtual NUMA controls (8.0)
• Virtual Topology in ESXi 8.0
• Hyperthreading with vSphere (8.0)
• CPU and memory statistics (Guest SDK, 7.0)
• Proxmox VE QEMU/KVM virtual machines
• Microsoft: CPU oversubscription and new CPU jitter counters in Windows Server 2025
• Microsoft: Detecting bottlenecks in a virtualized environment
• Microsoft: Hyper-V processor performance
• Microsoft: Manage Hyper-V scheduler types
• sys.dm_os_wait_stats
• sys.dm_os_schedulers
• Linux scheduler statistics
• Linux pressure stall information
• Linux control group v2
• Linux kernel sysctl: numa_balancing
• KVM paravirtual MSRs: steal time
• QEMU source: KVM vCPU thread name, Proxmox qemu-server source
• proc_pid_task(5), proc_pid_comm(5)
• proc_stat(5), vmstat(8), numactl(8), numastat(8)

This article was originally published on 10 November 2026.