A VM's Bulk Migration showed its data copied days ago. The switchover window opened, and the migration has not moved since. The HCX Manager log says the switchover is running in START_OFFLINE_SYNC.
Two things are easily mixed up here. Replication is the copying of the VM's disks, which happens over days before the window. Switchover is the short sequence inside the window: power off, a last sync, then build the VM at the target. A stall in switchover does not mean the replicated data is lost, and good replication figures do not mean the switchover can finish.
This article records the two separately, matches the stall to one documented HCX 4.11.3 cause, and hands the record over.
Scope
HCX 4.11.3 or later, Bulk Migration, a switchover stalled at START_OFFLINE_SYNC. Every step is a read. Not covered: Replication Assisted vMotion and other migration types, and any cancel, retry or cutover. Broader disaster recovery design has its own article: implement and test VMware-to-Azure disaster recovery.
The phases
KB 323663 describes a Bulk Migration in this order:
| Phase | What happens |
|---|---|
| Base sync | "a full sync event on the source vCenter Server starts replicating data to the target via the HCX-IX appliance" |
| Delta sync | "After completion of the initial base sync, an RPO cycle of 2 hours is set to perform the delta sync." |
| Power off | "HCX signals the source vCenter Server to power off the VM, which is required to stop further data churn on the source VM." |
| Offline sync | The last changes are sent and the replica disk files are consolidated |
| Instantiate | The VM's configuration files are copied and the VM is built at the target |
The first two are replication. The last three are switchover.
The documented cause (HCX 4.11.3 and later)
KB 438977 describes a switchover that "stalls at the START_OFFLINE_SYNC state" because "The source Virtual Machine was manually powered off prior to the HCX-orchestrated switchover phase." HCX needs the VM on so it can shut it down through VMware Tools; powering it off earlier "causing the Host-Based Replication (HBR) service to fail the final delta sync checkpoint". The KB names three log signatures, recorded in Step 3.
Step 1: record the migration
| Field | What to record |
|---|---|
| A | HCX Manager release at source and target, migration group, VM name, migration type |
| B | Switchover window as scheduled, or Switchover Now, with time zone |
| C | Current state shown in the HCX UI, and when it last changed |
Step 2: record replication progress
| Field | What to record |
|---|---|
| D | Base sync start and finish time, and size |
| E | Time of the last completed delta sync before the window |
| F | Any replication error or warning shown before the window |
Step 3: record switchover evidence
| Field | Where | What to look for |
|---|---|---|
| G | HCX Manager /common/logs/admin/app.log |
"Switchover running in state START_OFFLINE_SYNC", with time |
| H | vCenter events for the VM, or /var/log/vmware/vpxd/vpxd.log |
Power-off and guest shutdown events: time, and the user or tool that started them |
| I | Source ESXi /var/log/hostd.log |
Hbrsvc lines with "deltaState=lwd failed" and "removing group", with time |
| J | HCX-IX /var/log/vmware/hbrsrv.log and /var/log/messages.log |
Throughput figures and connection resets (KB 440867 comparator) |
| K | HCX Manager app.log |
"Can not poweroff VM as guest shutdown timed out and forcePowerOff is not set" (KB 321632 comparator) |
Record the exact text and times. Mask VM IDs and replication group IDs only if the record leaves the team.
Step 4: classify
| What the record shows | Candidate cause | Handed to |
|---|---|---|
| H shows a power-off before the switchover started, by a person or tool, and I shows "lwd failed" | KB 438977: VM powered off outside HCX | Migration owner, and the owner of that tool |
| J shows near-zero throughput or connection resets, and H shows HCX's own power-off | KB 440867 pattern: network performance. Broadcom notes migrations "may eventually complete after several hours" | Network owner |
| K appears | KB 321632: guest did not shut down in time; a different stage and error | Migration owner and VM owner |
| G only, with none of H to K | No documented match on this record | Migration owner, with the logs |
D to F establish which earlier replication stages completed. They do not verify the final delta checkpoint or the consistency of the target copy. The stall is recorded during switchover; final delta synchronisation and target consistency remain unverified.
What this record does not authorise
The record ends when it is handed over. It does not authorise cancelling, retrying, Switchover Now, Force Power-Off, powering the source VM on or off, cleaning up the failed job, or any network change. KB 438977's resolution includes cancelling and retrying; KB 323663 warns that cleaning up a failed job removes the seed data a retry could reuse. Those are the migration owner's decisions, under their own change record.
Worked example (fictional)
The values below are invented. HCX 4.11.3 at both sites, VM crm-app-02, Bulk Migration, window Saturday 22:00 AEDT.
| Field | Value |
|---|---|
| D | Base sync finished Wednesday 14:10, 820 GB |
| E | Last delta sync Saturday 18:02 |
| G | 22:00:41 "Switchover running in state START_OFFLINE_SYNC"; unchanged at 01:30 |
| H | 19:00 guest shutdown and power-off by the patching tool's service account |
| I | 22:00:44 "deltaState=lwd failed", 22:00:45 "removing group" |
| J | No resets, normal throughput before 19:00 |
Classification: KB 438977. The patching tool powered the VM off three hours before the window. An earlier delta sync completed at 18:02, before that shutdown. The 22:00 final delta checkpoint failed during switchover; the record does not establish that the target copy contains the final changes or is consistent. The record goes to the migration owner and the patching team. Nothing is cancelled or retried.
Acceptance checks
- Fields A to K are recorded, or marked not found with where you looked.
- Replication (D to F) and switchover (G to K) are recorded separately.
- Every power-off in H names who or what started it.
- Nothing was cancelled, retried, powered on or off, or cleaned up. Reading needs no rollback.
Learning notes
- Bulk Migration replicates first, then switches over inside the window.
- On HCX 4.11.3 and later, powering the source VM off before HCX does can stall switchover at START_OFFLINE_SYNC.
- The vCenter event shows who powered the VM off; that is usually the deciding line.
- Cleaning up a failed job can remove the copied data a retry would reuse.
Related articles
- Published: How to implement and test VMware-to-Azure disaster recovery
- Published: Scheduled jobs after migration: prevent duplicate work
Sources
Retrieved 2026-11-03 UTC (4 November, Australia/Sydney). Dates are each page's own Updated On date.
- Broadcom KB 438977, "Bulk Migration Hangs at Offline Sync Due to Manual Power Off - VMware HCX", 12 October 2026. https://knowledge.broadcom.com/external/article/438977/vmware-hcx-bulk-migration-hangs-at-offli.html
- Broadcom KB 323663, "HCX Bulk Migration operations and best practices", 26 June 2026. https://knowledge.broadcom.com/external/article/323663/hcx-bulk-migration-operations-and-best.html
- Broadcom KB 440867, "Slow migration performance and migrations stuck during the "Offline Sync"", 21 May 2026. https://knowledge.broadcom.com/external/article?articleNumber=440867
- Broadcom KB 321632, "HCX - Unable to Power Off Guest OS during Bulk Migration Switchover", 27 August 2026. https://knowledge.broadcom.com/external/article/321632/hcx-unable-to-power-off-guest-os-during.html