Map the slow application to its virtual disk and storage backend, then collect measurements at each boundary during the same workload. “The datastore is slow” is not a root cause: guest queueing, hypervisor queueing, physical devices, vSAN resynchronisation and network transport can all contribute.
This guide is diagnostic. It does not instruct you to delete snapshots, replace disks, change RAID layout or recreate disk groups. Those operations require a separate, validated recovery and maintenance procedure.
1. Describe the I/O workload
Record whether the operation is a small random read, sequential transfer, synchronous log write or metadata-heavy file workload. Capture block size, read/write mix, concurrency, working-set size and whether the application waits for durable completion.
A cached 128 MiB read test is not a measurement of sustained capacity-tier throughput. A high sequential bandwidth score does not predict a database log flush. Label every synthetic test accordingly and use only owned disposable files.
For an application incident, note the exact VM, guest disk/volume, virtual disk/controller, datastore and backing storage. In vSAN, also record storage policy, object health and whether the cluster uses Original Storage Architecture (OSA) or Express Storage Architecture (ESA). Do not apply an OSA cache-disk explanation to ESA storage pools.
2. Collect the guest and virtual-disk view
Measure guest read/write latency and queue length while replaying the operation. Collect per-disk information rather than only _Total when several virtual disks exist. For SQL, use interval file statistics from guide 5.
In vCenter, select the VM's virtual-disk performance charts for the same interval. Keep read and write latency separate, and record IOPS and throughput alongside latency. An apparent latency improvement after the application stops issuing I/O is not a successful fix.
Map each virtual disk to its backing datastore using VM settings. A guest drive letter is not a stable physical identifier. Include any snapshots, limits and storage policies in the map without modifying them.
3. Inspect ESXi device and path state
For a conventional block-storage datastore, use read-only inventory commands after checking installed help:
esxcli storage core device list
esxcli storage core path list
esxcli storage core device stats get --device naa.LAB_DEVICE_ID
Replace the placeholder with the exact identifier from the datastore mapping. Do not select a device merely because it is the largest or first in a list. The ESXCLI storage command reference: https://developer.broadcom.com/xapis/esxcli-command-reference/latest/namespace/esxcli_storage.html distinguishes inspection from claiming, unclaiming and modification commands.
Collect statistics twice around the workload and compare deltas where the fields are cumulative. Inspect path health and logs for resets, aborts or failed paths. A healthy-looking path count does not prove that all paths carry traffic as intended by the storage policy.
Do not change path selection or queue depth without storage-vendor support for that exact driver, firmware and array combination.
4. Interpret latency at the right layer
Use esxtop and its storage views for the relevant device. In the conventional device accounting model, DAVG describes device-side completion time, KAVG describes time within the kernel layer, and GAVG represents the combined observed storage latency. Interpret these for the selected object and transport, not as a universal decomposition of every vSAN/NFS operation. esxtop field reference: https://knowledge.broadcom.com/external/article/382249/using-esxtop-and-interpreting-esxtop-sta.html
| Pattern | Investigation |
|---|---|
| Device latency rises with workload | Backend contention, device/array response and fabric conditions |
| Kernel-side delay rises while device latency is modest | Queueing, host workload and supported queue limits |
| Guest latency rises without matching backend trend | Guest/application queueing, measurement scope and controller mapping |
| One device has outlier latency | Exact device/firmware/path health, not the datastore average alone |
| Latency rises while IOPS collapse | Timeouts, retries or stalled application, not improved efficiency |
Broadcom's “performance has deteriorated” log messages use microseconds and compare changes in device latency. Convert units before combining them with millisecond charts; a warning's magnitude and duration matter. Device-latency guidance: https://knowledge.broadcom.com/external/article/318927/performance-has-deteriorated-messages-in.html
5. Investigate vSAN frontend, backend and resync separately
In the cluster's Monitor > vSAN views, examine Performance, Skyline Health, Virtual Objects and Resyncing Objects for the test period. UI names may vary by update. Inspect client-visible latency alongside backend throughput, congestion and physical-disk observations.
Record which objects are affected and whether policy compliance has changed. If all affected VMs share a host, device or component placement, investigate that common boundary. Broadcom documents cases where physical-device latency manifests as VM latency: https://knowledge.broadcom.com/external/article/391147/latency-reported-for-virtual-machines-in.html; that is a diagnostic possibility, not proof for every incident.
Inspect free capacity and resynchronisation demand before running an additional benchmark. A rebuilding cluster is already performing essential recovery work. Do not add a sustained write benchmark simply to confirm that its disks are busy.
On the relevant host, these read-only commands help identify the vSAN network path:
esxcli vsan network list
esxcli network nic list
esxcli network nic stats get -n vmnic2
Map vmnic2 to the actual vSAN uplink. Guide 2 covers source-specific MTU testing; never assume the vMotion interface is also the vSAN interface.
6. Understand resynchronisation before changing its controls
A resync backlog can change as the system discovers and places work. Treat remaining bytes as a moving estimate, not a linear countdown. Correlate backlog, object health, throughput and application latency over time.
The repair-delay timer governs when some repairs begin; changing it does not accelerate an already running resync. Throttling resync trades reduced resource use against longer exposure to reduced redundancy. Neither is a default performance fix. Broadcom resync guidance: https://knowledge.broadcom.com/external/article/445909/reference-for-vsan-resync-wait-period-an.html
Document which objects are reduced-redundancy or non-compliant, the remaining fault tolerance and who accepts any extension of that period. Do not reduce protection policy to make a benchmark faster.
7. Validate the suspected cause with a narrow comparison
Use a lab workload that represents the application: same block sizes, concurrency, dataset and durability requirements. Record initial cache treatment and steady-state duration. Read-only tests still consume storage bandwidth; bound their runtime.
Examples of useful comparisons include the same disposable workload on two supported policies, before/after a vendor-approved driver fix, or an application run during and outside a naturally occurring resync window. Do not move production disks or alter a shared policy as an unreviewed experiment.
Keep host load and application demand in the report. A policy change can alter component layout and trigger resync, so measuring immediately after the change may measure migration work rather than the steady state.
8. Produce an escalation packet
Include UTC interval, application symptom, VM-to-device/object map, guest metrics, ESXi/vSAN charts, relevant device identifiers, firmware/driver versions and recent changes. Broadcom recommends checking the controller and drive support combinations when physical-disk latency is involved. vSAN disk-latency investigation: https://knowledge.broadcom.com/external/article/393495/high-disk-latency-on-vsan-cluster-may-im.html
Preserve complete relevant log context rather than only the highest latency line. Redact customer and infrastructure identifiers when sharing outside the approved support channel.
Acceptance and rollback
A change passes when the application's correct result and response target improve under comparable demand, while storage health and protection remain acceptable. Faster averages with new timeouts are a failure.
Restore the previous approved policy/configuration if that is a supported reversal, then allow any resulting resync to complete and verify compliance. Device replacement, consolidation and disk-group removal are not generic rollback mechanisms and are intentionally not supplied as executable repairs.
Related topics: SQL file-I/O diagnosis, VMware networking and backup recovery.
Learning notes
- Latency needs an object, a layer, a unit and an interval.
- Storage repair traffic and application traffic can compete for the same resources.
- Removing protection to improve a benchmark changes the requirement, not just performance.
Sources and review status
See the source ledger for dates, version scope and limits, the author self-check, and the series index. Independent review and lab validation remain pending.