Cutover runbook: replication lag, DNS TTL, validation gates and a real rollback

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A cutover plan needs evidence for four linked decisions: writes have stopped on the source, the target has reached a recorded position, DNS moves clients inside a window you calculated, and each stage has a stated rollback. Returning to an older source loses unreconciled target writes.

Identifiers and timings are illustrative.

Interpretations are editorial; no lab testing was performed.

Why it matters

Rollback must account for target writes. Zero lag and a short DNS TTL need context before they can support a cutover decision.

How it works

Lag: measure a position, not a feeling

A target can have received everything and still be applying it, and a time-based lag figure is an estimate. For a cutover, freeze writes, record the source's log position, and wait until the target has applied at least that position. That is the proof.

PostgreSQL

pg_stat_replication reports sent_lsn, replay_lsn, write_lag, flush_lag and replay_lag for each directly connected standby. An LSN (log sequence number) is a position in the write-ahead log, and pg_wal_lsn_diff returns the byte difference between two positions.

The time columns are not the proof. PostgreSQL says that when a standby has caught up and there is no more WAL activity, the last measured lag stays visible briefly and then shows NULL. With logical replication, a decoding plugin that emits no tracking messages also shows NULL. So NULL can appear in a healthy state and in an unmeasured one.

Use the position test after the freeze:

-- 1. On the primary, after all writers have stopped. Record the result.
SELECT pg_current_wal_lsn();

-- 2. On the standby. Replace the literal with the value from step 1.
SELECT pg_last_wal_replay_lsn(),
       pg_wal_lsn_diff(pg_last_wal_replay_lsn(), '0/5A000110') AS past_freeze_bytes;

For a confirmed physical standby of this source on the expected timeline, the position condition is past_freeze_bytes of zero or more. Verify that identity first; LSNs from unrelated histories are not comparable proof. Use "at least" and not "equal", because background activity can move the primary's position after your reading.

This test is for physical standbys only. pg_last_wal_replay_lsn returns NULL on a server started normally without recovery, which describes a logical replication subscriber. A marker row alone is not a complete logical-replication consistency proof. Use the version-specific subscription state and application boundaries, including parallel transactions and objects outside replication; this article does not define that procedure.

Confirm that the writers have really gone. pg_stat_activity has backend_type, client_addr and state. Only superusers and roles with pg_read_all_stats see every session's detail, so filter on backend_type = 'client backend' using one of those.

default_transaction_read_only sets only the default for each new transaction. Treat it as a guard against accidents and stop the applications as well. If you set it, the session writing the marker must first run SET default_transaction_read_only = off for itself.

Promotion uses pg_ctl promote or pg_promote(). pg_promote returns false if promotion did not finish within wait_seconds, so a script must check the result. At failover the standby restores WAL already available and makes no attempt to connect to the primary, so anything not received is left behind.

SQL Server Always On

Query sys.dm_hadr_database_replica_states on the primary, which reports every replica. Join sys.availability_replicas on replica_id if you want server names:

SELECT DB_NAME(database_id) AS database_name, replica_id, is_local,
       is_primary_replica, synchronization_state_desc,
       log_send_queue_size, redo_queue_size,
       last_commit_lsn, last_commit_time, secondary_lag_seconds
FROM sys.dm_hadr_database_replica_states
ORDER BY database_name, is_primary_replica DESC;

The fields mean different things:

A suspended secondary can report zero seconds of lag, so always read synchronization_state_desc beside it. Microsoft also warns that some values may be unavailable or out of date, and that they have only local relevance.

Our pass condition after the freeze: every database on the target shows SYNCHRONIZED, log_send_queue_size is 0, and last_commit_lsn on the secondary row equals the primary row.

A planned manual failover without data loss is supported only when both replicas are in synchronous-commit mode and the target is synchronised. Check is_failover_ready in sys.dm_hadr_database_replica_cluster_states, use the supported planned-failover procedure for that group type, validated in rehearsal.

The command returns once the target has accepted it, and database recovery continues asynchronously, so the next gate must check that databases are online. A read-scale group with CLUSTER_TYPE = NONE has its own failover procedure, so this command is not the method there. And consistency across several databases in the group "might not be maintained on failover", so check any application that spans databases.

SQL Server log shipping or manual log restores

There is no failover command here. Microsoft's controlled failover sequence is: copy any uncopied backups, apply them in order, take a tail-log backup, apply it, then recover the secondary. Microsoft notes you may need single-user mode for exclusive access before the restore. Its role-change procedure also says to disable the backup job on the original primary and the copy and restore jobs on the secondary, so include those jobs in the freeze.

The tail-log step is a planned outage: the NORECOVERY option leaves the source database in RESTORING and unavailable. Have the DBA implement the version-specific backup and restore procedure, with exclusive-access requirements, verified backup destinations and a rehearsed return path.

WITH NORECOVERY guarantees the database does not change after the tail-log backup. It also keeps the way back open, because Microsoft says you can later roll this database forward with log backups from the replacement primary. Microsoft also tells you to make the secondary's metadata consistent with the original primary when you bring it online.

The pass condition here is that every log backup, including the tail, restored without error and in order, and that the database came online after recovery with the marker row present. A secondary held in NORECOVERY cannot be read before that.

Whole-VM replication

Machine-level tools report their own lag, and it does not say whether the application was quiet.

AWS Transform MGN describes Lag as how far behind an instance launched now would be (the page says "recovery instance"), and Backlog as data written at the source and not yet copied. Its readiness indicators are the lifecycle state Ready for cutover and a Healthy replication status. All four are the pass condition.

Azure Migrate agentless replication runs an on-demand final delta when you migrate. Its concepts page says you must shut down the on-premises VM first to prevent data loss; the tutorial makes that a choice for "a planned migration with no data loss". Treat it as required. Complete Migration will also "clean up resources and shut down the source virtual machines", so do not run it while the source is your way back. The tutorial's test migration is the natural rehearsal.

Where a database inside the VM is replicated separately, use the database proof and treat the machine's lag as secondary evidence.

DNS: the TTL you must wait out is the old one

A resolver can keep an answer for the TTL it received. Lowering the TTL affects only answers fetched after the change. A resolver that fetched the record just before you lowered it keeps the old answer for the full old TTL.

RFC 1034 states the method: reduce the TTL before an anticipated change, then raise it afterwards.

Query each authoritative server directly, because a recursive answer tells you only about that resolver's cache:

dig @ns1.example.net app.example.com A +noall +answer

Caches that do not behave like the TTL table

Negative answers. A resolver may cache "this name does not exist" for the lower of the SOA MINIMUM field and the SOA record's TTL. Create new names days ahead.

Serve-stale. RFC 8767 lets a resolver use expired data for a suggested 1 to 3 days when authoritative servers are unreachable. Do not change DNS provider in the same window.

Java. InetAddress has its own cache, controlled by networkaddress.cache.ttl. In Java 21 the default is forever with a security manager and implementation-specific without one, which is the usual case today. So set it explicitly, or time a test name change in rehearsal. Failed lookups use networkaddress.cache.negative.ttl, default 10 seconds. If in doubt, restart Java services after the change.

Existing connections. A connection pool that stays open keeps using the old address however low the TTL was, which is one reason to fence the source.

What to choose and why

Decide rollback by stage, before the night

The way back depends on where the newest committed data lives. Write this table for your own system.

Stages B and C are not the same action on every path. This table is our reading of the sources and has not been lab tested.

WITH RECOVERY has a cost. Once recovered, the source can no longer take log backups from the new primary, so replication must be seeded again before another attempt.

Stage E is the point of no return in practice. If losing those writes is unacceptable, your plan has to end the rollback option before clients start writing, or build the reverse path first.

Reverse paths differ by technology.

• Always On: after failover, the former primary becomes a secondary and synchronises from the new primary. Keep it connected and failback is another planned failover.
• Log shipping: the source left in RESTORING by the tail-log backup can receive log backups from the new primary. Set that job up before opening to users.
• PostgreSQL: pg_rewind can bring the old primary back as a standby of the new one. It needs a cleanly shut down target, full_page_writes on, either wal_log_hints or data checksums, and WAL reaching back to the point of divergence. PostgreSQL warns that if it fails while processing, the target's data folder is "likely not in a state that can be recovered", so back up the old primary first or be ready to rebuild it. Configuration files are copied from the source, so correct the recovery settings before restarting.
• MGN and Azure Migrate: the pages reviewed describe no replication from target back to source. MGN's Revert to "ready for cutover" changes the lifecycle status only, and after finalising MGN stops replication. Treat VM-level rollback as stage E once clients write, unless the database has its own reverse path.

Validation gates

A gate is a named person, a measurement, a pass value and an abort action. Appoint one person to decide. If a gate has no result by its deadline, stop progression and invoke the pre-agreed recovery decision. Do not automatically restart an older source after target writes have begun.

A marker is a useful identity and path check, not a complete consistency proof: after writers stop, insert one uniquely identified row on the source, then read it on the target. It catches replication of the wrong database or schema, which a healthy lag figure cannot.

Where gate 1 runs depends on whether the target can be read:

• PostgreSQL hot standby, or a readable Always On secondary: read the marker before takeover. A failure is stage B. Readability is configured per replica, so confirm it.
• Non-readable Always On secondary: confirm the position from the primary before failover, and read the marker after it. A marker failure is stage C.
• Log shipping: insert the marker before the tail-log backup, because that backup is what carries it, and read it after recovery. Stage C.
• Whole-VM: the target does not exist until launch. Insert the marker before the source is shut down, and read it after launch. Stage C.

The runbook

Times are illustrative. Replace them with your rehearsal measurements.

The fence comes before promotion, so that no client with a stale answer can write to the old source after the target has taken over. On rollback, reverse the order: fence the target first, confirm it, then release the source.

The fence must not cut replication. A PostgreSQL standby connects with primary_conninfo, normally on the client port, so stopping the listener stops replication. Restrict by client address. Always On replicates through a separate database mirroring endpoint, which must stay open. A blocked port may leave established sessions running. The DBA's freeze procedure must account for them without interrupting replication or active transactions unexpectedly.

A DNS rollback obeys the same arithmetic, so keep the TTL low until gate 5 has closed.

Risks and limits

• Fencing failure. If a client can write to both systems, you have two divergent databases. We know of no general way to merge them automatically.
• Non-database state. Files, message queues, caches and scheduled jobs on the source are not covered by database lag. Add each to gate 2.
• Limits of this article. PostgreSQL statements come from the version 18 documentation. MySQL and other engines are not covered.

Do not delete the source or finalise in the migration tool until the close gate has passed, and do not begin a cutover whose rollback has never been rehearsed.

Sources

Opened 2026-11-10.

• PostgreSQL: Monitoring Database Activity
• PostgreSQL: System Administration Functions
• PostgreSQL: Log-Shipping Standby Servers
• PostgreSQL: pg_ctl
• PostgreSQL: pg_rewind
• PostgreSQL: Client Connection Defaults
• Microsoft: sys.dm_hadr_database_replica_states
• Microsoft: Monitor performance for availability groups
• Microsoft: Perform a planned manual failover
• Microsoft: Tail-log backups, BACKUP, Recover without restoring data
• Microsoft: Readable secondary replicas, Database mirroring endpoint, KILL
• Microsoft: Fail over to a log shipping secondary, Change roles between log shipping servers
• AWS Transform MGN: Migration lifecycle
• AWS Transform MGN: Ready for cutover indicators
• Microsoft: Agentless replication of VMware VMs
• Microsoft: Migrate VMware VMs agentless

• RFC 1034, RFC 2308, RFC 8767

• Oracle: Java 21 networking properties

• dig(1)

This article was originally published on 10 November 2026.