Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

Azure Site Recovery: reading replication health, RPO breaches and failed test failovers

Technical article
Cloud

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Azure Site Recovery gives you three separate signals, and each answers a different question. Replication health says whether data is flowing fast enough. Recovery point objective (RPO) says how old your recoverable data is. A completed test failover is the only one of the three that shows a machine can actually start in the other region.

A green health state does not prove recovery will work, and a failed test failover often has nothing to do with replication. Read them separately.

This article covers Azure VMs replicating between Azure regions. The queries and paths come from Microsoft's published documentation and were not run in a lab for this article.

Why it matters

Disaster recovery is usually judged at the worst possible moment. A vault can show healthy machines for months while a recovery network has been deleted, a subscription has run out of quota for the target region, or the domain controller the application needs was never part of the plan.

The opposite mistake also costs time. An RPO alert prompts someone to examine the network when the real cause is a nightly job writing faster than the replica disk can absorb.

Microsoft recommends a test failover at least once every six months. It also defines success narrowly: a test failover counts only after both the failover and the post-failover cleanup have completed.

How the data moves

Knowing the stages tells you where a delay can sit. For Azure VMs, the replication process runs as follows:

  1. The Mobility service extension is installed on the VM.
  2. Disk writes transfer immediately to a cache storage account in the source region.
  3. Site Recovery processes the cached data and sends it to the replica managed disks in the target region.
  4. Crash-consistent recovery points are generated every five minutes, a frequency you cannot change.
  5. App-consistent points are created at the frequency set in the replication policy.

A crash-consistent point captures on-disk data as if the power had been pulled. An app-consistent point uses application coordination, including Volume Shadow Copy Service (VSS) on Windows, to capture a consistent on-disk state. This does not mean that a recovered VM resumes its running memory image.

The default replication policy keeps recovery points for one day, and does not take app-consistent snapshots at all. That default deserves attention on any database server. The minimum app-consistent frequency is one hour.

The fine-grained points do not last. Site Recovery prunes recovery points after two hours, saving one point per hour, so the five-minute choices exist only for the most recent two hours.

Replication needs outbound connectivity only. Site Recovery never needs inbound connectivity to the VM.

Reading the health state

The states in Replicated items are:

State Meaning
Healthy Replication is progressing normally, with no error or warning symptoms
Warning One or more symptoms that might affect replication
Critical Replication error symptoms, typically replication stuck or not keeping pace with the data change rate
Not applicable Not expected to be replicating, for example after failover

Two other parts of the dashboard answer questions health does not.

Configuration issues tests whether you could fail over. A validator runs every 12 hours by default, and the refresh icon beside the section heading forces it to run now. It reports missing settings, missing resources such as a deleted target virtual network, and subscription quota. Software update notices appear in the same section, but that validator does not produce them. The quota check compares what is available against what failing over every machine in the vault would need, covering VM cores, VM family cores and network interface counts.

Error summary counts how many machines each active symptom affects. Microsoft points out that this count helps show when one underlying issue affects many machines. A single symptom across every VM in a vault points away from any one guest. That reading is editorial.

Two different numbers: RPO and latest recovery point

The monitoring FAQ separates them:

  • RPO is calculated once changes and the tag identifying a recoverable point have been written to storage in the target region. It equals the time elapsed since that point's timestamp.
  • Latest available recovery point is later in the pipeline. It is the newest point already processed and applied to the replica disks, and so usable at failover.

Expect the two to differ. A low RPO alongside an older latest recovery point suggests processing on the target side is behind upload. That inference is editorial, so confirm it against the events on the machine.

One more check before believing an RPO figure. Microsoft states that incorrect system time on the source machine skews the computed RPO. If the source clock jumps forward and back quickly, a separate time-change problem can appear. Microsoft describes waiting until system time passes the skewed future time, or a disable-and-re-enable procedure for forward replication only. The latter is a protection change requiring a reviewed recovery-point and reprotection plan; it is not a routine diagnostic step.

Tracing an RPO breach

When Site Recovery cannot create recovery points for 60 minutes, it raises error ID 153007: "No crash consistent recovery point available for the VM in the last 60 minutes." Microsoft's troubleshooting article deals first with a high data change rate, then with network connectivity. Checking in that order is a sensible default.

Step 1: look for a churn event

Churn is the rate at which disks change. Go to Replicated items, then the VM, then Events - last 72 hours, and look for "Data change rate beyond supported limits". Selecting the event identifies the disk.

Errors show what is wrong now. Events are the history of what affected the machine, so a resolved spike appears only there.

Step 2: measure the write rate

On the source VM, chart OS Disk Write Bytes/Sec and Data Disk Write Bytes/Sec. These show the combined pattern, and may not give per-disk detail.

If logs are flowing to a Log Analytics workspace, Microsoft publishes a query for churn against upload rate. Replace the machine name:

AzureDiagnostics
| where Category in ("AzureSiteRecoveryProtectedDiskDataChurn", "AzureSiteRecoveryReplicationDataUploadRate")
| extend CategoryS = case(Category contains "Churn", "DataChurn",
    Category contains "Upload", "UploadRate", "none")
| extend InstanceWithType = strcat(CategoryS, "_", InstanceName_s)
| where TimeGenerated > ago(24h)
| where InstanceName_s startswith "<vm-name>"
| project TimeGenerated, InstanceWithType, Churn_MBps = todouble(Value_s)/1048576
| render timechart

Reading the shape is editorial. Churn above upload with RPO climbing means more is arriving than can leave. Churn well below upload with RPO still climbing points away from churn and toward the network or the service.

Step 3: compare with the limits

There are two limits, per disk and per VM. The per-disk figure depends on the replica disk and the average write size:

Replica disk Average source I/O Churn per disk
Standard storage 8 KB 2 MB/s
Premium SSD 128 GiB or more 8 KB 2 MB/s
Premium SSD 128 GiB or more 16 KB 4 MB/s
Premium SSD 128 GiB or more 32 KB or greater 8 MB/s
Premium SSD 512 GiB or more 8 KB 5 MB/s
Premium SSD 512 GiB or more 16 KB or greater 20 MB/s

Microsoft notes these limits are based on tests and that results vary with the application's I/O mix. Normal Churn supports up to 54 MB/s per VM.

A short burst is not necessarily a problem. Microsoft says that if the rate exceeds 10 MB/s for Premium or 2 MB/s for Standard and then falls, replication will catch up. The concern is churn that stays well above the limit.

If it does stay high, the documented options include excluding the disk, which first requires disabling replication, or increasing the size of a Premium replica disk. Resizing helps only if that disk's churn is below 20 MB/s, or below 50 MB/s with High Churn.

Microsoft's resize steps are to copy the replica disk name from the replicated machine's Disks page and open that managed disk. If a banner says a SAS URL has been generated, select it and cancel the export. Then increase the size under Size + Performance.

Our inference, which Microsoft does not state, is that a larger replica has a higher base tier, so the resize can trigger the BookmarkNotFound condition described later. Expect to wait for a new recovery point after resizing before relying on the recovery position.

Increasing the replica's performance tier will not help. Microsoft states the churn limit follows the replica's disk size and is unchanged by tier. A 128 GiB replica moved to P50 keeps its 128 GiB limit.

The other route is High Churn, which uses a Premium Block Blob cache account:

VM memory Maximum churn per VM Maximum per disk
Less than 32 GB 54 MB/s 20 MB/s
32 GB to less than 256 GB 100 MB/s 50 MB/s
256 GB and above 500 MB/s Up to 250 MB/s, subject to disk size

High Churn limits also depend on write size, operating system, region pair, Mobility service version and cache account. On Linux, churn above 100 MB/s is supported only for RHEL 9, SLES 15 and Ubuntu 24.04, and 500 MB/s needs Premium SSD v1, Premium SSD v2 or Ultra Disk source disks. All four Australian regions appear in Microsoft's list. Microsoft also warns that High Churn might lead to higher network costs. High Churn can be chosen only when enabling replication. Switching a protected VM means disabling replication and enabling it again.

Step 4: test the network path

Latency to the cache becomes a problem if uploading is slower than 4 MB in 3 seconds. Microsoft suggests testing the upload with AzCopy from the VM. If latency is high, check whether a network virtual appliance is controlling outbound traffic, as it may be throttled when all replication passes through it. The documented remedy is a virtual network service endpoint for Storage.

Where outbound access is controlled by URL, these destinations are required:

Purpose Commercial cloud URL
Cache storage *.blob.core.windows.net, or the specific cache account URLs
Authentication login.microsoftonline.com
Site Recovery service *.hypervrecoverymanager.windowsazure.com
Monitoring and diagnostics *.servicebus.windows.net

App-consistent points failing

Error ID 153006 means no app-consistent point has been created in the past X minutes. Crash-consistent points may still be arriving, so the recovery position may be better than the alert implies, though without application consistency.

On Windows, open this file on the source machine and search for vacpError:

C:\Program Files (x86)\Microsoft Azure Site Recovery\agent\Application Data\ApplicationPolicyLogs\vacp.log

Microsoft's published error codes include:

Code Meaning Documented action
2147221164 "VSS writer is not installed" (the text discusses the Site Recovery VSS Provider) Follow the VSS writer installation guidance
2147943458 "VSS writer is disabled" (the text discusses the provider service) Set it to Automatic, restart VSS, the provider and VDS
2147754756 Provider not registered Reinstall using the agent's uninstall and install scripts

Other documented causes include SQL Server instances with AUTO_CLOSE databases, and Storage Spaces Direct, where app-consistent points cannot be created. Linux needs custom pre and post scripts. Shared disks support crash-consistent points only.

Getting alerted

Built-in Azure Monitor alerts cover critical replication health and failover failures. They are generated by default, but you must create an alert processing rule before they send notifications. The same page documents a separate email subscription for critical events. A vault with neither route can show alerts that nobody receives.

For trend data, send the vault's diagnostics to a workspace, selecting the Azure Diagnostics toggle and all logs prefixed AzureSiteRecovery. Churn and upload data arrive every five minutes, and replication statistics and recovery points every 15 minutes, so very short events can be missed.

This is Microsoft's RPO trend query:

AzureDiagnostics
| where replicationProviderName_s == "A2A"
| where TimeGenerated > ago(72h)
| where isnotempty(name_s) and isnotnull(name_s)
| where name_s == "<vm-name>"
| project TimeGenerated, name_s, RPO_in_seconds = rpoInSeconds_d
| render timechart

Reading a failed test failover

A test failover does not affect ongoing replication or production, provided it goes to an isolated network. For Azure VMs, Microsoft's drill tutorial starts it from the VM's Overview page, with a choice of Latest processed, Latest, Latest app-consistent or Custom, and advises a non-production target network. Track it on the Jobs tab, which shows stages. The stage that fails tells you where to look:

Stage What it does Failure points to
Prerequisites Checks conditions for failover Configuration issues on the machine
Failover Processes data so a VM can be created Data or recovery point problems
Start Creates the Azure VM Quota, capacity, policy, network resources

The right-hand column is editorial, based on what each stage does.

Failure creating the VM

Microsoft's failover troubleshooting page is written for VMware and physical machines. Its causes are general Azure conditions, but confirm the applicability for Azure VMs. Error IDs 28031, 28092, 70038 and 170010 are listed under causes that include:

  • Insufficient quota for VMs or network interfaces. Check Subscription, then Usage + quotas.
  • VMs of different size families in the same availability set.
  • A subscription policy that blocks VM creation.

"One or more replica IP addresses is invalid"

The cause given is that cleanup of a previous test failover did not happen, so a test machine remains and holds the address. Complete the cleanup and retry.

A failure after a disk tier or SKU change

If a disk's tier or SKU changed, a failover using a recovery point from before the change eventually fails with BookmarkNotFound. Older points may still be listed until pruning removes them. Wait for a recovery point created after the change.

The VM started but the application did not

Microsoft's VMware and physical server page gives this order, so confirm it fits Azure VMs:

  1. Check Boot diagnostics for a screenshot of the VM.
  2. If the VM did not start, try an older recovery point.
  3. If the application is down, try an app-consistent point.
  4. If domain-joined, confirm that a domain controller is working, by testing a domain join from a new VM in the same network.
  5. If custom DNS is in use, confirm name resolution from a new VM in that network.

Active Directory needs planning before a drill. For a test failover, you need a domain controller inside the isolated network. It should own the Flexible Single Master Operations roles the test requires, otherwise the Active Directory owner must plan the role requirements and review any role-seizure procedure for an isolated drill. Do not seize roles in a network that can contact production. The test network's DNS should use the address expected for the DNS VM, and site-to-site connectivity should not be enabled on it.

What to choose and why

These are editorial recommendations.

Situation Action
Choosing a recovery point in a drill Use the point you would use in a real event. Latest has the lowest RPO, but Microsoft's on-premises failover page says log processing stops if the source region goes down and you must use the latest processed point. We have not found that statement on an Azure VM page, so practise Latest processed too
Database server Enable app-consistent points deliberately, and test the failure path for 153006
Applications spanning VMs Consider a replication group, noting it affects performance and blocks single-VM failover
Sustained churn above the limit Size the replica disk, then expect to wait for a new recovery point, or plan High Churn. Do not change the performance tier
Drill scope Test the recovery plan with its dependencies, not one VM

Use a test network with the same subnet names and address range. If no subnet of that name exists, the VM lands in the first subnet alphabetically, which can break a dependency unnoticed.

Risks and limits

Warning: do not cancel a real failover once it is running. Microsoft's on-premises failover page says replication stops before failover starts, and that if the job is cancelled, the VM does not replicate and replication cannot be started again. We found no equivalent statement on an Azure VM page, so treat it as applying until Microsoft says otherwise.

Warning: committing a failover deletes all recovery points held by the service, and removes the Change recovery point option. Validate the VM first.

Warning: a test failover into a production network is disruptive. The primary VM must be shut down or two VMs with one identity run on the same network, and the application is down during the test.

  • Any change made to a test VM is lost at cleanup and is not replicated back.
  • Do not cancel or skip the cleanup. It removes the test VMs, and an incomplete cleanup causes the IP error above.
  • Disabling replication, whether to exclude a disk or to change churn type, stops protection until it is re-enabled. Allow for that exposure window. That caution is editorial.
  • Soft delete is not supported on the cache storage account.
  • If Site Recovery cannot create new points beyond the retention window, old points remain until new ones replace them. Old points existing does not mean replication is working.
  • Churn limits come from Microsoft's tests. Treat them as guidance.
  • VMware and Hyper-V replication differ in components and error handling, and this article does not cover them.

Sources

Opened 10 November 2026. Sources behind the corrections reopened 10 November 2026.