Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

ESXi: Device performance has deteriorated. I/O latency increased from average value

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

vmkernel.log on an ESXi host contains WARNING: ScsiDeviceIO: ... Device naa.########## performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds. (Broadcom KB 318927). The host is reporting that a storage device answered much more slowly than usual. The numbers are microseconds and correspond to DAVG in esxtop. Broadcom's KB says "ESXi does not cause latency spikes; it merely reports them", so the fault is in the device, the storage network, the array or the load.

What the message means

Broadcom's KB 318927 says the numbers "are in microseconds and refer to the DAVG measurements in the esxtop storage screen". It defines latency as "the round-trip time between the issuance of a SCSI command from the hypervisor, through the transport to the surface of the media, and the return". The source of a delay "could be anywhere in the fabric, the storage infrastructure, or anywhere along the storage path."

The KB lists when the message appears: the latency ratio is "significantly higher than previous logging", and has doubled since the previous log entry. A matching performance has improved message is logged when latency falls again.

The same page gives two benchmarks for DAVG:

Storage type Benchmark in Broadcom's KB In microseconds
Non-flash about 10 milliseconds 10,000
Flash "rare to see DAVG latency above 1-2 milliseconds" 1,000 to 2,000

Broadcom's KB 344099 defines the esxtop columns. DAVG/cmd is "The average response time in milliseconds per command being sent to the device". KAVG/cmd is "The amount of time the command spends in the VMkernel". GAVG/cmd is the response time "as it is perceived by the guest operating system", and "DAVG + KAVG = GAVG". It says the three "should not exceed more than 10 milliseconds (ms) for sustained periods of time", and that if the response time is over 5000 ms, "ESXi will time out the command and abort the operation."

Some NFS datastores log a different line, NFS volume ... performance has deteriorated. KB 335047 describes it for NFSv3 from ESXi 7.0 and NFSv4.1 from ESXi 7.0 Update 1, with a default threshold of 10 ms. This guide covers the SCSI device message.

NetLeverage runs VMware vCenter, vSphere and ESXi.

Likely causes

Broadcom's KB 318927 lists changes to the target, disk or media failures, overload conditions and failover events as causes, and tells you to check scheduled tasks and supported HBA drivers. KB 335047 adds the network path for NFS. None can be read from the message alone.

Cause What you see How to confirm
Overload on the device Spikes at busy times, high DAVG esxtop device view shows high DAVG/cmd with high CMDS/s
Scheduled tasks such as backup or replication Spikes at fixed times Message timestamps line up with job schedules
Changes made to the target, or a failover event One spike after a known change The array or fabric log shows the change at the same time
Disk or media failure on the array Repeated spikes, sometimes on one device Failures logged on the storage array
Unsupported HBA driver Spikes on devices behind one adapter Driver version against the Broadcom Compatibility Guide
Network path fault (iSCSI or NFS) Spikes with no array load Port errors on the switch or SFP, as KB 335047 describes for NFS

Prove the cause before changing anything

All of these are read-only.

1. List the messages. KB 318927 gives the log as var/run/log/vmkernel.log (under / on the host):

grep -E 'performance has (deteriorated|improved)' /var/run/log/vmkernel.log

Write down each device ID, the two figures, and the time. Pair each "deteriorated" line with the "improved" line for the same device. KB 318927 says to match the device ID and to confirm that the "latency increased" to value matches the "latency improved" from value.

2. Find the datastore on that device. KB 316540 gives these commands:

esxcli storage vmfs extent list
esxcli storage core device list

The first maps each datastore to its device name. The second lists the devices currently connected to the host.

3. Watch live latency. KB 344099 describes this procedure: run esxtop, then press u for the disk device view, d for the disk adapter view, or v for the per-VM view. Pressing s then 2 and Enter sets a two-second update. Read DAVG/cmd, KAVG/cmd and GAVG/cmd.

A good result is DAVG/cmd under 10 ms on non-flash and about 1 to 2 ms on flash, and the figures in the log lines are single short events. A bad result is DAVG/cmd high for long periods. If DAVG/cmd is large and KAVG/cmd is small, then GAVG = DAVG + KAVG means almost all the delay is beyond the host. That points to the array, the fabric or the path.

4. Characterise the spikes. KB 318927 gives four questions: magnitude (how high is DAVG), duration (how long each spike lasts), frequency (the pattern of the timestamps) and scope (one datastore or several, one host or several).

5. Check the schedule. KB 318927 says to check whether the messages "are generated during scheduled tasks (e.g., backups or replications)".

Fix

The fix sits outside ESXi, which is why no command is given for it. Match the action to the finding:

  • Consistently high latency. KB 318927: "check the storage performance. If failures are logged on the storage array side, contact the storage vendor for further assistance."
  • Spikes at scheduled times. KB 318927 says to check whether the messages are generated during scheduled tasks, and gives no remedy for this case beyond checking the schedule.
  • Overload. KB 318927: "reduce the load on the affected storage device."
  • Driver. KB 318927 says to verify that the host runs supported drivers for the HBA, using the Broadcom Compatibility Guide for the ESXi version.
  • Network path (iSCSI or NFS). KB 335047 says to check switch ports for incrementing CRC errors, input errors or runts, and SFP diagnostics. It notes that a single faulty SFP can cause periodic packet loss that looks like datastore latency in the logs.

If devices disconnect, or virtual machines freeze or have filesystems marked read-only, the problem is past latency. The guide "APD or PDL: classify an inaccessible ESXi datastore" covers how to tell the cases apart. The guide "How to diagnose VMware and vSAN storage latency" covers the path from the guest to the backend.

Example (illustrative): esx01 logs Device naa.600000000000000000000000000000a1 performance has deteriorated. I/O latency increased from average value of 4210 microseconds to 52630 microseconds. at 02:10:05, and an "improved" line for the same device at 02:24:35. The arithmetic: 4210 microseconds = 4.21 ms, and 52630 microseconds = 52.63 ms. The spike is 52.63 / 4.21 = 12.5 times the average. It lasted 02:24:35 - 02:10:05 = 14 minutes 30 seconds, which is 870 seconds. esxcli storage vmfs extent list maps that device to ds-prod01. In esxtop, DAVG/cmd is high and KAVG/cmd is low at that hour. The nightly backup of the VMs on ds-prod01 starts at 02:00. The administrator asks the storage vendor to check the array for the same hour and compares the backup schedule with the log times.

Confirm the fix

  1. After the change, grep -E 'performance has (deteriorated|improved)' /var/run/log/vmkernel.log shows fewer or no lines for that device.
  2. In esxtop, DAVG/cmd for the device stays under the benchmark for its storage type at the busy hour.
  3. The vCenter storage latency alerts stop for that datastore.
  4. Guest-side I/O wait falls.

What to watch afterwards

  • Keep the log lines from the incident: device ID, figures, times. They answer the four questions under "Characterise the spikes".
  • Compare the next month's counts of "deteriorated" lines per device.
  • After any change on the array or fabric, read the log for the hour that follows.

Risks and limits

  • These commands have not been run or lab-tested.
  • Broadcom's KB 318927 was updated 11-17-2026 and KB 344099 on 03-23-2026. The 10 ms and 1 to 2 ms figures are Broadcom's rules of thumb, and Broadcom's KB 344099 says "all arrays perform differently". Compare them with your array vendor's figures.
  • The message identifies a device and a time. It does not identify a cause.
  • esxtop shows a live view only. Spikes between your samples can be missed, so use the log lines to find the times.
  • Broadcom's KB 318927 gives the log sample for SCSI devices only. The text of the "improved" line for a SCSI device is not shown on that page, so search for the phrase, not a full line.

Sources