Diagnose a slow VM by combining application timings with guest and host measurements from the same interval. Adding vCPUs or memory without identifying the constrained layer can consume capacity while leaving the application unchanged.
This guide provides read-only collection and a controlled lab experiment design. VM resizing, reservations and host-wide power changes are separate configuration operations; no production tuning has been validated here.
1. Define the symptom and observation window
Choose an actual transaction: open a report, execute a representative query or complete an application login. Record the input, expected output, concurrent users, data size and elapsed time. Use synthetic or approved lab data.
Write down VM identity, vCPU count, assigned RAM, virtual hardware version, guest version, host placement and any resource-pool membership. Capture CPU/memory reservations and limits from VM Edit Settings. A host with spare CPU can still contain a VM restricted by a limit.
Synchronise clocks or record measured clock offsets. A minute of guest counters and an unrelated host average cannot establish causality. Capture at least a normal period and a reproducible slow period; a short lab sample does not establish peak-day capacity.
2. Collect a bounded host sample
Use an authorised ESXi shell. Interactive esxtop provides CPU and memory views; c selects CPU, m memory, f chooses fields and q exits. A bounded batch collection is useful when a workload is being replayed:
esxtop -b -a -d 5 -n 60 > /vmfs/volumes/LAB-DATASTORE/diag/esxtop.csv
Create a unique approved output directory beforehand and check free space. The example collects about five minutes and writes a potentially large CSV; -a increases collection overhead. Do not schedule repeated full collections without a retention plan. The command flags and field definitions are documented in Broadcom's esxtop reference: https://knowledge.broadcom.com/external/article/382249/using-esxtop-and-interpreting-esxtop-sta.html.
3. Retrieve existing vCenter samples
In an authenticated, single-vCenter PowerCLI session, resolve one VM and request available real-time data:
$vm = @(Get-VM -Name 'APP-LAB-01' -Server $vc)
if ($vm.Count -ne 1) { throw 'Expected one exact lab VM' }
$metrics = @(
'cpu.usage.average', 'cpu.ready.summation',
'cpu.costop.summation', 'mem.vmmemctl.average',
'mem.swapped.average', 'mem.swapinRate.average',
'mem.swapoutRate.average'
)
Get-Stat -Entity $vm[0] -Stat $metrics -Realtime -MaxSamples 30 -Server $vc |
Select-Object Timestamp,MetricId,Instance,Value,Unit,IntervalSecs |
Export-Csv './APP-LAB-01-host-samples.csv' -NoTypeInformation
This retrieves retained samples; it does not sleep while collecting 30 new ones. Schedule the query after the bounded workload or explicitly specify the desired time window. Unsupported metrics are a collection gap, not zero activity. Confirm actual interval, unit and instance from each result. Get-Stat reference: https://developer.broadcom.com/powercli/latest/vmware.vimautomation.core/commands/get-stat
4. Calculate CPU Ready correctly
CPU Ready measures time a virtual CPU could run but was waiting to be scheduled. A summation in milliseconds is not a percentage. For a per-vCPU sample:
ready_percent = ready_milliseconds / (sample_seconds × 1000) × 100
For an aggregate containing all vCPUs, divide by the vCPU count only when you intentionally want the average per-vCPU percentage. Do not divide a per-vCPU sample again.
Hypothetical example: 1,200 ms over 20 seconds is 6% for one vCPU. If 1,200 ms is the aggregate across a four-vCPU VM, the average is 1.5% per vCPU. That average can hide one hot vCPU, so inspect the individual instances as well. Broadcom explains the difference between Ready representations: https://knowledge.broadcom.com/external/article/387750/understanding-the-cpu-ready-values-in-th.html.
Use sustained patterns and workload response, not one universal threshold. A short burst, historical rollup or VM-wide sum can mislead a threshold-only diagnosis.
5. Separate four CPU cases
| Pattern | Interpretation to investigate | Controlled next step |
|---|---|---|
| Guest CPU saturated; low Ready | Workload needs execution capacity or more efficient code | Profile hot work; test one sizing change |
| High Ready across several VMs | Host scheduling contention or placement constraints | Inspect host demand and DRS/resource policy |
High %MLMTD |
CPU limit is restricting eligible execution | Review the intentional limit with its owner |
| Co-stop with oversized SMP VM | Multi-vCPU scheduling and topology may be inefficient | Test a smaller VM against identical work |
%WAIT is not a direct measure of wasted CPU: it includes states that need interpretation, including idle behaviour. Do not subtract arbitrary counters to manufacture a contention score.
Treat host power policy as a shared-host variable. A guest High Performance test does not prove the effect of changing ESXi or BIOS power management.
6. Separate guest pressure from hypervisor reclamation
Collect guest available memory, commit usage/limit, working sets and page-read/write activity. In Windows, hard page faults may represent loading executable or mapped-file pages; page-ins alone do not prove pagefile thrashing.
At the hypervisor layer, compare balloon size, swapped memory and swap rates. Ballooning is a reclaim mechanism; the important question is whether the guest has to discard useful pages or page heavily under the workload. Nonzero historical swap occupancy and current swap traffic are different observations.
If a guest process leaks memory while host ballooning and swapping remain zero, address that process. Adding host swap-to-SSD cache does not repair an application memory leak. Conversely, a guest with apparently adequate assigned RAM can suffer when a host reclaims heavily. Keep the layers separate in the report.
Do not disable balloon drivers or set blanket memory reservations as a diagnostic shortcut. Reservations consume admission capacity and can affect other VMs.
7. Inspect NUMA and virtual topology before resizing
Record physical NUMA-node capacity and the VM's exposed topology. A VM larger than one node may legitimately be a wide VM; forcing it into one node is not always possible or desirable.
For ESXi 8.0 Automatic vTopology, Broadcom identifies hardware version 20 or later and the “Assigned at power on” cores-per-socket setting as relevant conditions. It also describes behaviour that differs from older blanket advice about CPU Hot Add disabling vNUMA. Check the exact VM hardware and configuration before applying an inherited rule. ESXi 8.0 rightsizing guidance: https://knowledge.broadcom.com/external/article/438023/rightsizing-virtual-machines-on-esxi-80.html
Do not upgrade virtual hardware simply to follow this article. Compatibility and downgrade/recovery implications require a separate plan.
8. Run one controlled sizing experiment
Use a disposable clone with representative data and no production integrations. Record the original CPU/RAM configuration and the business acceptance criteria. Perform A/B/A runs: original, one changed setting, original again. Shut down through the guest where the change requires it.
Keep application version, dataset, concurrency and cache treatment consistent. Record warm-up separately. Compare median and tail response, throughput, errors, Ready and guest paging. If the second baseline moves substantially, investigate host load or cache drift before crediting the change.
A useful result states the workload, scope and uncertainty: “Four vCPUs reduced this lab transaction's tail latency under this concurrency.” It does not say “all Windows VMs should use four vCPUs.”
Acceptance and rollback
Accept a candidate only when the application output remains correct and the measured improvement repeats without exhausting failover headroom. Restore the original CPU/RAM configuration if it fails. Preserve both successful and failed measurements.
If evidence points to a shared-host bottleneck, prepare a separate placement or hardware-policy change with capacity review. This guide does not authorise changing every VM or rebooting the cluster.
Related topics: Windows application performance, SQL waits and storage latency.
Learning notes
- Ready milliseconds need an interval and an instance before they become a percentage.
- Guest paging, ballooning and host swapping describe different layers.
- More virtual hardware is useful only when the workload benefits from it.
Sources and review status
See the source ledger for dates, version scope and limits, the author self-check, and the series index. Independent review and lab validation remain pending.