About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
When a Linux VM logs watchdog: BUG: soft lockup - CPU#0 stuck for 22s!, the kernel's watchdog found that one CPU had not let any other task run for longer than its limit. On a physical server that points at a kernel or hardware bug. On a VM it may equally mean the host did not give the virtual CPU any time. Broadcom's guidance is to investigate the ESXi host first: CPU contention, storage delays, snapshots and migrations.
Do not raise the watchdog threshold to make the message stop. That hides the stall and leaves its cause in place.
What the symptom means
The Linux kernel documentation defines a soft lockup as "a bug that causes the kernel to loop in kernel mode for more than 20 seconds ... without giving other tasks a chance to run". By default "the system will stay locked up", and the stack trace of the current task is printed. A hard lockup is the stricter case: a CPU that loops "without letting other interrupts have a chance to run".
The 20 seconds comes from a tunable. watchdog_thresh defaults to 10 seconds, and "The softlockup threshold is (2 * watchdog_thresh)", so 2 × 10 = 20 seconds. The detector runs from a timer that fires every 2 × watchdog_thresh ÷ 5 = 4 seconds, so a message that says 21 or 22 seconds is what you would expect just past the limit. Broadcom's KB 313465 shows stuck for 22s and stuck for 21s in its ESXi guest samples, and KB 412869 shows stuck for 21s. In the current kernel source the line is BUG: soft lockup - CPU#%d stuck for %us! [%s:%d], printed with the prefix watchdog: , followed by the CPU number, the seconds and the name and process ID of the running task.
Older kernels and older Broadcom pages differ. Broadcom's KB 328790, for older kernels, shows BUG: soft lockup detected on CPU#1!, says many kernels report when the watchdog thread is not scheduled for more than 10 seconds, and points at softlockup_thresh. That is a different default and a different tunable from the current documentation.
Why a VM is different
Broadcom's KB 328790 puts the difference this way: "On a physical host, a soft lockup message generally indicates a kernel bug or hardware bug. When running in a virtual machine, this might instead indicate high levels of overcommitment (especially memory overcommitment) or other virtualization overheads." KB 447755 adds: "If the hypervisor delays giving CPU time to the VM for too long, the internal watchdog timers in the Linux kernel expire. The guest OS interprets this hypervisor-induced pause as a frozen process".
The duration in the message is how long the CPU was stuck, so the stall began about that many seconds before the log timestamp.
Likely causes
| Cause | What you see | How to confirm |
|---|---|---|
| Host CPU contention or oversized VM | Several VMs slow at once; CPU ready or co-stop high | esxtop CPU view (c): %RDY, %CSTP, %MLMTD; vSphere CPU ready chart |
| Storage delays | Messages with disk or network file system activity; device latency up | esxtop disk view; performance has deteriorated lines in the host logs |
| Duplicate IP on a storage VMkernel port | Several Linux VMs on one host report soft lockups together; NFS latency up | vobd.log duplicate IP events; vmkernel.log ARP lines |
| Snapshot with quiesce | Message starts as a snapshot is created or removed | Snapshot task time against the message; VM stun times |
| vMotion of a busy VM | Message during or just after a live migration | vMotion task time against the message |
| Fault Tolerance with VMCI DMA datagrams (ESXi 8.0 to 8.0U2) | FT-protected Linux VM on hardware version 20 or later, whose Linux kernel has the VMCI DMA datagrams feature, locks up | VM compatibility (hardware version 20 or later), FT status, ESXi build, and a guest kernel and distribution on Broadcom's list (below) |
| A real guest deadlock | Stuck time keeps climbing into tens of thousands of seconds (Broadcom's sample reaches 143406s); VM unresponsive | Sample times in the log; kernel and product version |
Prove the cause before changing anything
On the guest, read only:
journalctl -k
sysctl kernel.watchdog_thresh
sysctl kernel.softlockup_panic
journalctl -k shows kernel messages. Note each message's time, CPU number, stuck seconds and task name. kernel.watchdog_thresh shows the limit in force. kernel.softlockup_panic is 0 or 1; the kernel documentation says 1 means "Panic on soft lockup", which turns the warning into a guest crash and reboot, as Broadcom's KB 313465 notes for some supported distributions when kernel.softlockup_panic is set to 1.
On the host, read CPU first. In esxtop press c. Broadcom's esxtop overview (KB 308290) lists %RDY at 10 or more ("If values are between five and ten percent take care"), %CSTP at 3 or more, and %MLMTD at 1 or more ("percentage of time a ready to run vCPU was not scheduled because of a CPU limit setting"). KB 447755 says "%RDY values consistently above 5% to 10% per vCPU are abnormal". KB 387750 warns that "The CPU ready or %RDY in itself is not an indicator for existing performance issues". The thresholds differ between KB 308290 and KB 447755, so read them as a range. The units also differ: KB 308290's esxtop figure is summed over the VM's vCPUs, while KB 447755 says "per vCPU" (KB 387750 explains the sum, so divide by the vCPU count to compare).
Charts need arithmetic. Broadcom's article on the vSphere Client charts explains that ready time is a total in milliseconds over the chart interval, "20 seconds apart" in Real-Time, and that it is the sum across all virtual CPUs, so "the value displayed in the chart must be divided by the number of virtual CPUs".
Then read storage: in esxtop press u for the disk device view (KB 447755), and the host logs for performance has deteriorated. For storage network faults, Broadcom's example shows vobd.log lines Duplicate IP address detected for [192.0.2.1] on interface vmk# and vmkernel.log lines arp: [..] is using my IP address [192.0.2.1] on vmk2!.
Example (illustrative): app01 has 8 vCPUs and logs watchdog: BUG: soft lockup - CPU#3 stuck for 22s! at 14:05:30, so the stall began about 14:05:08. The Real-Time CPU ready chart shows a summed ready value of 8,000 ms in a 20-second sample. As a percentage that is 8,000 × 100 ÷ 20,000 = 40% across all vCPUs, and 40 ÷ 8 = 5% per vCPU. Five percent is the edge of Broadcom's cautionary range, not proof by itself. The host log for the same minute shows a snapshot removal on app01, so the first check is the stun time for that task.
Fix
Cause: host CPU contention
Reduce the load or the size: move other VMs, remove CPU limits that %MLMTD shows, and size vCPUs to the work. Broadcom's overview says to decrease the vCPU count when %CSTP is high and to remove a CPU limit when %MLMTD is 1 or more. See "CPU overcommit in hosted clouds: ready time, NUMA and how to size honestly" and "How to find and fix ESXi CPU and memory bottlenecks".
Cause: storage delays or a duplicate storage IP
For latency, see "How to diagnose VMware and vSAN storage latency". For the duplicate address, Broadcom's resolution is that "the duplicate IP conflict must be cleared at the network layer": find the MAC address named in vobd.log, locate the switch port that learned it, and "Change the IP address of the conflicting device or the affected VMkernel interface to a unique value."
Disruptive step: changing the IP address of a VMkernel storage interface can interrupt datastore access on that host. Do it in a planned window. Back out by setting the previous address again, unless that address was the duplicate.
Cause: snapshots and vMotion
For snapshot-related stalls, see "My VM freezes for a while when a snapshot is created or removed". For vMotion, Broadcom's page for a SAP HANA VM says the hypervisor may not schedule the guest for a prolonged time "due to a resource contention issue on a host or complications during a vmotion", and gives advice such as "Use vMotion only during non-peak times (for example, when CPU utilization is less than 25%)". That advice is for a SAP HANA VM; apply it to other busy VMs as your own judgement.
Cause: Fault Tolerance with VMCI DMA datagrams
Broadcom's article states the cause as "The VMCI DMA datagrams feature is incompatible with Fault Tolerance", fixed in "ESXi 8.0 U3 build 24022510". Broadcom says the VM is affected only when all of these hold: it is protected by Fault Tolerance, it is on virtual hardware version 20 or later, and the Linux kernel has the VMCI DMA datagrams feature. Broadcom says the feature is supported from upstream Linux 5.18 and lists the distributions as RHEL 8.7 and 9.1 and later, Ubuntu 22.04.1 LTS (kernel 5.15.0-41.44 onwards) and 22.10, SLES 15 SP3 (kernel 5.3.18-150300.59.93.1 onwards) and SLES 15 SP4, and Photon 4.0. Non-FT VMs, FT VMs older than hardware version 20 and FT VMs without the feature are not impacted. Broadcom's workaround for ESXi 8.0, 8.0.1 and 8.0.2 starts by verifying the hardware version and guest OS version. Only then power off the FT VM, add the attribute vmci.dmaDatagramSupport with value FALSE, and power it on.
Disruptive step: this requires powering off the FT VM, which takes its service down while it is off. Do not do it to a VM that fails the conditions above. Back out by removing the attribute and powering the VM on again.
Raising or switching off the watchdog
KB 170185 tells the reader to write a "new_timeout_value" to /proc/sys/kernel/watchdog_thresh and calls it "the desired time in seconds before a soft lockup is triggered". The kernel documentation says the soft lockup threshold is 2 × watchdog_thresh, so the page's reading of the value as the seconds to a soft lockup does not match the documentation; use the 2 × rule below.
Risk: Broadcom says raising the timer "is generally not recommended as it can mask underlying issues and should only be used for troubleshooting purposes in a controlled environment." The kernel documentation adds that setting watchdog_thresh to zero "will disable lockup detection altogether". If you must test, note the current value, for example sysctl -w kernel.watchdog_thresh=15 (a soft threshold of 2 × 15 = 30 seconds), and write the old value back to undo.
Cause: a real guest deadlock
Broadcom's page for one VMware product stack shows watchdog: BUG: soft lockup - CPU#4 stuck for 143406s! lines on an unresponsive VM that could not be pinged, and traces it to a kernel spinlock deadlock fixed in a later kernel build. Stuck times in the tens of thousands of seconds, a VM that does not answer at all and a quiet host point at the guest kernel. Update the kernel with the vendor, and keep the full log for the vendor.
Confirm the fix
- No new soft lockup messages over a period that includes the busy time of day and any snapshot or backup windows.
esxtopCPU and disk figures sit inside the range your team accepts, and no newperformance has deterioratedlines appear.- The application on the VM reports no timeouts or failovers in the window where messages used to appear.
What to watch afterwards
- Ready time and co-stop per VM, as a percentage of the sample interval and per vCPU.
- Snapshot and migration schedules that overlap with busy periods.
- FT and ESXi build if you use Fault Tolerance on ESXi 8.0 releases before 8.0 U3.
- Panic setting. Keep
kernel.softlockup_panicat the value you chose on purpose.
Read next
Related articles you may want next.
- PerformanceCPU overcommit in hosted clouds: ready time, NUMA and how to size honestlyMeasure CPU wait on ESXi, Proxmox VE and Hyper-V, read ready time correctly, keep VMs inside NUMA boundaries and size vCPUs from evidence.Read article
- PerformanceHow to find and fix ESXi CPU and memory bottlenecksCorrelate application latency with CPU scheduling, limits, NUMA, ballooning and swap activity.Read article
- TroubleshootingMy VM freezes for a while when a snapshot is created or removedLearn what a VM stun is, read stun times from vmware.log, find what makes snapshot pauses long, and assess version-specific change options and their application impact.Read article
Risks and limits
- These commands have not been run or lab-tested. They come from the kernel documentation, the kernel source, Broadcom articles and the
sysctlmanual page listed below. - Kernel behaviour changed over time. The tunables and message text here are for current kernels. Older kernels use
softlockup_threshand shorter limits, as Broadcom's older pages show. - Broadcom's CPU ready thresholds differ between KB 308290 and KB 447755, and KB 387750 says ready time on its own is not an indicator of a performance issue.
- A single message cannot tell a host stall from a guest bug. If the host is quiet and the numbers keep climbing, involve the guest operating system vendor with the full log.
Sources
- Softlockup detector and hardlockup detector (aka nmi_watchdog) — Linux kernel documentation, development version shown 7.4.0-rc3, accessed 18 November 2026
- Documentation for /proc/sys/kernel/ (watchdog_thresh, softlockup_panic) — Linux kernel documentation, accessed 18 November 2026
- kernel/watchdog.c, master — Linux kernel source, no date shown, accessed 18 November 2026
- Error: "kernel: BUG: soft lockup - CPU#Y stuck for Xs" within VM (KB 170185) — Broadcom, updated 07-20-2026, accessed 18 November 2026
- Soft lockup messages from Linux kernel running in an SMP-enabled virtual machine (KB 328790) — Broadcom, updated 01-13-2026, accessed 18 November 2026
- Understanding the CPU ready values in the vSphere Client advanced performance charts (KB 387750) — Broadcom, updated 05-14-2026, accessed 18 November 2026
- ESXTOP overview for Performance Troubleshooting (KB 308290) — Broadcom, updated 11-26-2025, accessed 18 November 2026
- Virtual machine hangs due to RCU stall and sapstartsrv soft lockup on RHEL guest (KB 447755) — Broadcom, updated 11-03-2026, accessed 18 November 2026
- Troubleshooting Linux Guest OS "soft lockup" errors due to ESXi Storage VMkernel IP conflicts (KB 442986) — Broadcom, updated 06-15-2026, accessed 18 November 2026
- SAP Hana VM reporting soft CPU lockups and TX hangs during a vmotion (KB 412869) — Broadcom, updated 10-21-2025, accessed 18 November 2026
- vSphere Fault Tolerance protected Linux VMs on HWv20 and later with the VMCI DMA datagrams feature may lockup or panic on ESXi8.0/8.0U1/8.0U2 (KB 313465) — Broadcom, updated 09-23-2026, accessed 18 November 2026
- Error: "watchdog: BUG: soft lockup - CPU## stuck for #####!" (KB 397009) — Broadcom, updated 06-24-2026, accessed 18 November 2026
- journalctl(1), sysctl(8) — man7.org, no date shown, accessed 18 November 2026
This article was originally published on .