Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

USB passthrough device disconnects from the VM

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

ESXi can pass a USB device plugged into the host through to one VM. TechDocs: "A USB device is available to only one virtual machine at a time." A host service called the USB arbitrator does the work: "It scans the host for USB devices and manages the device connection among virtual machines that reside on the host." When the device drops from the VM, the cause is usually one of three things. The arbitrator lost the device or never fully owned it. The device type is not passed through. Or something the host or VM did, such as a migration, suspend or hot-add, disconnected it as designed. The logs and a few commands tell you which.

What the error means

There is often no error at all. KB 413940 describes a device that is added successfully, then "within few seconds, the USB disappears with no errors." The trace is in the VM's vmware.log, which that KB shows "flooded" with lines like:

USB: Device [name:Microchip\ Hub\ Controller ... path:0/1/1/3 speed:high ...] should already be connected

On the host, /var/run/log/usb.log records what the arbitrator excluded. KB 409283 shows: Excluding device from enumeration because it is marked a hub. In the vSphere Client, KB 387417 shows the add itself failing: Cannot connect "path:#/#/#" to this virtual machine. The device was not found.

TechDocs lists disconnects that are expected behaviour. "Hot adding these resources disconnects USB devices, which might result in data loss." "During the suspend or resume process, USB devices behave as if they have been disconnected, then reconnected." And if the arbitrator is stopped, "USB devices attached to that host become unavailable to the virtual machine."

Likely causes

Cause What you see Source
Smart card reader claimed by host services Disappears in seconds; "should already be connected" KB 413940
Device detected as a hub usb.log "marked a hub"; passthrough disabled KB 409283
Device class not passed through HID boot devices, isochronous audio/video, CCID, hubs KB 315272
Moved to another host port Autoconnect does not reconnect TechDocs autoconnect
Hub or extension cable Intermittent drops TechDocs connecting
VM suspended or powered off after vMotion Device cannot reconnect TechDocs vMotion
DPM powered off the device's host Device drops after DPM action TechDocs vMotion
CPU, memory or PCI hot-add Device disconnects TechDocs data loss
Arbitrator stopped All passthrough devices gone; /etc/init.d/usbarbitrator status reports it stopped ESXCLI reference, TechDocs data loss

Prove the cause before changing anything

These steps only read. Use the ESXi shell over SSH.

1. List the host's USB devices and their passthrough state. KB 409283 and KB 398992:

lsusb
esxcli hardware usb passthrough device list

The ESXCLI reference describes the second as "List usb devices and their passthrough status". A device with passthrough disabled will not stay with the VM. KB 398992 also prints lspci | grep -i usb to check the host's controllers.

1a. Check the arbitrator is running. The ESXCLI reference for esxcli hardware usb passthrough device enable says a device can be connected to the VM "only if usbarbitrator is running" and gives the check: /etc/init.d/usbarbitrator status. If it reports stopped, every passthrough device on the host is unavailable to its VM, which matches TechDocs' "USB devices attached to that host become unavailable to the virtual machine."

2. Read the host's USB log. Look in /var/run/log/usb.log for "Excluding device from enumeration" lines naming your device.

3. Read the VM's vmware.log. Search for lines starting USB: around the time of the drop. Repeated "should already be connected" lines match KB 413940's case.

4. Check the device class. KB 315272 lists what is not passed through: "USB devices such as mouse and keyboards that have a bootable HID interface.", "USB devices such as real-time video cameras and audio devices that use isochronous data transfers.", "USB CCID (chip/smart card interface devices)." and "USB hub." It adds: "The devices not in the following list are available for passthrough including USB storage devices, USB dongle devices etc."

5. Check the VM's hardware. KB 398992: "Virtual machines require both a USB controller and USB passthrough device configuration to access physical USB devices connected to the ESXi host." TechDocs: "A controller must be present before you can add a USB device to the virtual machine."

6. Ask what changed. Was the device moved to another port? TechDocs: autoconnect "uses the physical topology and port location instead of the device identity", and "If you plug the same device back into a different USB port on the host, it cannot reestablish the connection with the virtual machine." Was the VM migrated, then suspended? Is the host in a DRS cluster with DPM? Was CPU or memory hot-added?

Example (illustrative): a licence dongle moved from the host's rear port to a front port during maintenance stops appearing in its VM. esxcli hardware usb passthrough device list shows the dongle with passthrough enabled, at a new path. The VM's USB entry still points at the old path. Autoconnect follows the port, not the device, so the fix is to put it back or re-add it.

Fix

Device moved or swapped. Plug it back into the original port, or remove and re-add it in the VM's Edit Settings. TechDocs also warns that "The USB port is speed-specific", so a replacement device at a different speed may not autoconnect.

Hubs and cables. TechDocs: "Chaining hubs together also increases the chance of port and hub error, which can cause the device to lose connection to a virtual machine." "Connecting certain USB devices directly to the host rather than to a hub or extension cable might resolve their connection or performance issues." It adds: "In some cases, you must hard reset the device and hub to restore the device to a working state."

Device detected as a hub (modem case). KB 409283 enables it by ID with esxcli hardware usb passthrough device enable -d <busID>:<devID>:<vendorID>:<productID>, then sets a quirk:

esxcli system settings advanced set -o /USB/quirks -s 0x<vendorID>:0x<productID>:0:0xffff:UQ_NET_IGNORE

Disruptive step: KB 409283 then says "Reboot the ESXi host"; move or power off its VMs first. The way back for the passthrough change is esxcli hardware usb passthrough device disable -d <busID>:<devID>:<vendorID>:<productID>, documented in the ESXCLI reference. The KB gives no undo for the /USB/quirks setting, so ask Broadcom support before removing it.

Smart card readers and security keys. KB 413940 stops the arbitrator and smart card daemon, powers off the VM, adds three advanced options, adds the device, powers on the VM, then starts the services again:

/etc/init.d/usbarbitrator stop && /etc/init.d/pcscd stop
usb.generic.allowCCID = "TRUE"
usb.generic.allowHID = "TRUE"
usb.generic.allowLastDevice = "TRUE"
/etc/init.d/usbarbitrator start && /etc/init.d/pcscd start

The three usb.generic lines go in "VM Options" > "Advanced" > "Edit Configuration". KB 413940 also says: "Before configuring USB in a VM, ensure VMtools are updated." KB 387417 adds only usb.generic.allowCCID = "TRUE" for a security key, with the VM powered off. Disruptive step: stopping the arbitrator disconnects every passthrough device on that host, and TechDocs says restoring it by restarting services "requires that you power off and then power on the virtual machine."

vMotion, suspend and DPM. TechDocs: after a VM with USB devices is migrated, "if you suspend or power off the virtual machine, the USB devices are disconnected and cannot reconnect when the virtual machine is resumed. The device connections can be restored only if you move the virtual machine back to the host to which the devices are attached." For DPM: "deactivate DPM for that host. Otherwise DPM might turn off the host with the attached device." Every passthrough device must be enabled for vMotion or "the migration cannot proceed."

Hot-add. TechDocs: "Before you hot add memory, CPU, or PCI devices, you must remove any USB devices." Risk: a USB storage device disconnected mid-write can lose data.

Unsupported class. For a device on KB 315272's list, use another method, such as a network-attached device or a supported reader. A USB CD-ROM drive is a separate case. KB 387502: "By design ESXi will emulate USB CDROM device as SCSI CDROM device and USB passthrough is disabled". Its fix, written for ESXi 7.0 only, sets a vmkusb module parameter and needs a host reboot:

localcli system module parameters set --module=vmkusb --parameter-string="usbCdromPassthroughEnabled=1"

After the reboot it verifies with localcli system module parameters list --module=vmkusb and esxcli hardware usb passthrough device list. Disruptive step: plan the reboot as for any host maintenance. KB 387502 documents no rollback.

Confirm the fix

  • esxcli hardware usb passthrough device list shows the device with passthrough enabled.
  • No new "Excluding device" lines in usb.log.
  • No repeating USB: lines in the VM's vmware.log.
  • The guest sees the device after a VM restart and, where used, after vMotion.

What to watch afterwards

  • Physical changes to host ports during maintenance.
  • DPM settings on hosts with USB devices.
  • Hot-add requests on VMs with USB devices.
  • Host reboots and patches, after which the arbitrator re-scans devices.

Risks and limits

  • KB 413940 is written for one smart card reader model; follow it only for CCID devices.
  • TechDocs: "The USB arbitrator can monitor a maximum of 15 USB controllers. Devices connected to controllers numbered 16 or greater are not available to the virtual machine."
  • On resume, a Linux guest "might mount the USB devices at a different location on the file system."
  • TechDocs' feature table marks vSphere DPM and vSphere Fault Tolerance "No" with USB passthrough, and DRS and vMotion "Yes".

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources