Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Linux VM on VMware: "INFO: task blocked for more than 120 seconds" and what to check on the host

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When a Linux guest logs INFO: task <name>:<pid> blocked for more than 120 seconds., the kernel's hung-task detector has found a task that has been waiting, without running, for at least two minutes. The message is a symptom report, not a diagnosis. On a virtual machine one reason is that an I/O request the task is waiting for has stalled, so the useful next step is to line the time of the message up with storage latency, snapshot activity and controller type on the ESXi host.

The next line, "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message., only silences the warning. It does not unblock the task.

What the symptom means

The kernel source shows what the detector does. In the current development source the warning is printed as INFO: task %s:%d blocked%s for more than %ld seconds., where the optional text is in I/O wait. At tag v6.19 the same line has no optional text. The number printed is (jiffies - t->last_switch_time) / HZ in both: the seconds since the task last ran, so you can see values above 120 if the stall continues. The kernel documentation says "When a task in D state did not get scheduled for more than this value report a warning" and "0 means infinite timeout, no checking is done."

A task in D state is in uninterruptible sleep. The ps manual page defines D as "uninterruptible sleep (usually IO)". The kernel configuration help for the default says "A timeout of 0 disables the check. The default is two minutes", which is the 120 you see. Distribution kernels can set another value, so read it from /proc/sys/kernel/hung_task_timeout_secs on the guest.

Other facts from the kernel documentation that change how you read the logs:

  • hung_task_panic defaults to 0, "Continue operation". A non-zero value makes the kernel panic when that many hung tasks are found in one scan.
  • hung_task_warnings is a budget of detailed warnings. "When this budget reaches 0, no more detailed warnings will be reported", and it is reset when no hung task is found. A quiet log after a long stall does not prove the stall ended.
  • hung_task_detect_count "Indicates the total number of tasks that have been detected as hung since the system boot", so it counts what the log may not show.

Likely causes

Cause What you see How to confirm
Storage latency or a stalled request on the host Blocked tasks show block-layer or file system frames in the call trace; several VMs on the same datastore complain esxtop disk view latency (DAVG, KAVG, GAVG), performance has deteriorated lines in vmkernel.log
Snapshot or quiesce activity Stall begins as a snapshot is created or removed Snapshot task times in vSphere against the first message
Virtual NVMe disk with an older Linux kernel on ESXi 7.0.3 nvme messages about a timeout, aborting and resetting a controller before the hung-task lines Kernel log around the first message; Broadcom's article 318038
Host CPU or memory contention Guest slows generally; hung-task messages appear with soft lockup messages CPU ready on the host; see "Linux VM on VMware: "watchdog: BUG: soft lockup - CPU#0 stuck for 22s" and what to check before touching the guest"
A wait inside the guest on a lock Call trace names a mutex or semaphore, sometimes the task that holds it The trace text in the log; newer kernels can print the owner

Prove the cause before changing anything

On the guest, read only:

journalctl -k
dmesg -T
cat /proc/sys/kernel/hung_task_timeout_secs
ps -eo pid,stat,wchan:32,comm

journalctl -k shows kernel messages only (-k is the same as --dmesg). dmesg -T gives human-readable timestamps, and its manual page warns the timestamp "could be inaccurate", so correlate by the time in journalctl as well. In ps output, D in the STAT column is the state the detector watches, and the wchan column names the kernel function the task is waiting in. Read the call trace beneath the message: block-layer, file system or nvme frames point at storage, and a mutex or semaphore frame points at a wait inside the guest.

On the ESXi host, run esxtop, press u for the disk device view (or v for per-VM disk), and watch the latency columns. Broadcom defines them: DAVG/cmd is "The average response time in milliseconds per command being sent to the device", KAVG/cmd is "The amount of time the command spends in the VMkernel", and GAVG/cmd is "The response time as it is perceived by the guest operating system" calculated as DAVG + KAVG = GAVG.

Broadcom's pages do not agree on the numbers to worry about, so read them as a range, not a rule:

  • KB 344099 says DAVG, KAVG and GAVG "should not exceed more than 10 milliseconds (ms) for sustained periods of time";
  • the esxtop overview (KB 308290) lists DAVG at 25 or more, KAVG at 3 or more, GAVG at 25 or more, and Resets/s or ABRTS/s at 1 or more;
  • KB 447755, on a guest hang, treats DAVG "above 20ms or spikes to 2,000ms+" as a storage or fabric issue.

KB 344099 says "If the response time is over 5000 ms, ESXi will time out the command and abort the operation", with the events logged in /var/log/vmkernel.log. KB 447755 shows lines of the form Device naa.<id> performance has deteriorated. I/O latency increased from average value of <n> microseconds to <n> microseconds. and gives the log path as /var/run/log/vmkernel.log, so look in both places. The number is in microseconds, so 776,315 microseconds is 776.3 ms (776,315 ÷ 1,000).

Example (illustrative): db01 logs INFO: task jbd2/sda2-8:312 blocked for more than 120 seconds. at 02:10. esxtop was not running, but vmkernel.log on esx01 shows performance has deteriorated. I/O latency increased from average value of 12000 microseconds to 450000 microseconds for the datastore's device at 02:08. That is 12 ms rising to 450 ms (450,000 ÷ 1,000), a 37.5-fold increase (450,000 ÷ 12,000). Two other VMs on the same datastore logged the same message. The cause is on the storage path, so the fix is not inside db01.

Fix

Cause: storage latency or an aborted request

Treat it as a storage incident. Preserve the first error in the guest and the host logs, then work down the path with the storage owner. Broadcom's guidance for sustained high device latency is to investigate switches and SAN targets and "Check for errors that may suggest a delay in operations". See "How to diagnose VMware and vSAN storage latency" and "APD or PDL: classify an inaccessible ESXi datastore", and for a file system that has already gone read-only, "A Linux Filesystem Became Read-Only: Preserve the First Error".

Cause: snapshot or quiesce

Compare snapshot task times to the first message. Broadcom lists "The VM is undergoing a snapshot with quiesce" among the possible causes of soft lockup messages in a guest (KB 170185). That page is about soft lockups, not the hung-task message, so treat it as a reason to test snapshot timing, not as proof. See "My VM freezes for a while when a snapshot is created or removed" for how to read stun times in the VM's log.

Cause: virtual NVMe on older kernels (ESXi 7.0.3)

Broadcom's article describes a hang when an ABORT command times out, seen on Ubuntu 18.04 (kernel 4.15) and RHEL 8.3 (kernel 4.18) guests with a vNVMe disk, with a sequence of kernel lines in its sample (I/O 42 QID 2 timeout, aborting, Abort status: 0x0, then timeout, reset controller and timeout, disable controller; the numbers vary) followed by the 120-second message. The documented cause is "Bugs in the earlier Linux NVME Drivers when using the vNVME new Retry Handler, under APD or with heavy IO load conditions", and Broadcom states the issue "has been fixed in 7.0 P05 (u3f)". It also says an advanced setting exists to turn off the vNVMe retry subsystem, without naming it. If this matches, raise it with whoever manages the host patch level.

Cause: contention or a lock inside the guest

If the trace shows a lock and the host is quiet, the fix belongs to the application or kernel owner. Keep the full trace.

Silencing the message

Risk: writing 0 to hung_task_timeout_secs stops the detector, and raising it delays the warning. Neither fixes the stall, and both remove the alert. Do not do this to make a log quiet. If you set it for a controlled test, record the old value first; write the old value back to undo it.

Confirm the fix

  1. No new blocked for more than lines after the change, and hung_task_detect_count stops increasing.
  2. esxtop latency columns sit below the figures your storage team accepts, and no new performance has deteriorated lines appear in vmkernel.log.
  3. The task named in the message completes its work; the application checks pass.

What to watch afterwards

  • Latency trend. Graph device latency in vSphere, and alert before the guest's 120 seconds, not after it.
  • Aborts and resets. The esxtop overview lists Resets/s and ABRTS/s at 1 or more as a signal. Check them after storage changes.
  • The warning budget. If the log stops after a few warnings, use hung_task_detect_count to see whether tasks are still being caught.
  • Patch levels. Keep the vNVMe case in mind when a guest kernel and ESXi build are both old.

Risks and limits

  • These commands have not been run or lab-tested. They come from the kernel documentation, Broadcom articles and manual pages in the Sources list.
  • The kernel text and defaults come from the development source at 18 November 2026 and tag v6.19. Older enterprise kernels print the same core line, as Broadcom's 4.15 sample shows, but the optional text and the exact configuration differ.
  • Broadcom's latency thresholds differ between KB 344099, KB 308290 and KB 447755. None is a guarantee.
  • Network file systems, multipath settings and guest-side disk timeouts are outside what is covered here.
  • A single message does not identify its cause. Use several VMs, both log sets and timing together.

Sources