Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

How long will a full backup take? Working out the backup window from data size and link speed

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A full backup takes the data size divided by the rate of the slowest stage between the source disk and the backup store. Link speeds are quoted in bits per second, so divide by 8 to get bytes per second, then allow for the share of the link you will actually get. Compare the answer with the hours you have each night. When it does not fit, a full every night is impossible, and that is one reason incremental backups exist.

The figures here cover the backup direction only. How long the copy takes to come back is a different measurement.

Which units must you get right?

Three habits cause wrong answers.

  • Bits and bytes. Network links are sold in bits per second (Mbit/s, Gbit/s). Data and disks are measured in bytes. One byte is 8 bits, so 1 Gbit/s is 1,000,000,000 / 8 = 125,000,000 bytes per second, or 125 MB/s.
  • Decimal and binary. NIST gives 1 MB = 10^6 bytes = 1,000,000 bytes, and 1 MiB = 2^20 bytes = 1,048,576 bytes. A tebibyte (TiB) is therefore about 10% larger than a terabyte (TB): 2^40 / 10^12 = 1.0995. A disk reported as 8 TiB holds about 8.8 TB (NIST, prefixes for binary multiples).
  • Line rate and achievable rate. A 1 Gbit/s link will not carry 125 MB/s of backup data. Protocol overhead, other traffic, and a slow disk at either end all lower it. The example below uses an assumed 70%; replace it with a measured figure.

What is the formula?

time in seconds = data size in bytes / rate in bytes per second

Rearranged, the rate you need to fit a window is size / window. The table shows the best case, at line rate with no overhead. Treat each figure as a floor that real backups exceed.

Data to copy (decimal) 100 Mbit/s (12.5 MB/s) 1 Gbit/s (125 MB/s) 10 Gbit/s (1,250 MB/s)
1 TB 80,000 s = 22.2 h 8,000 s = 2.2 h 800 s = 13.3 min
8 TB 640,000 s = 177.8 h (7.4 days) 64,000 s = 17.8 h 6,400 s = 1.8 h

Check one cell yourself: 8 TB is 8 x 10^12 bytes; at 125 x 10^6 bytes per second the time is 64,000 s, which is 17.8 hours.

Which stage is the slowest?

The rate in the formula is the slowest of several stages, and the link is only one of them.

  1. Reading the source. The disk, or the hypervisor, may supply data more slowly than the link can carry it.
  2. Processing. Veeam describes an active full backup as retrieving the data "from the source datastore", then compressing and de-duplicating it before it is written (Full Backup Methods). That takes processor time, and it also means the bytes sent over the link can be fewer than the bytes read.
  3. The network. The part the table above measures.
  4. Writing the store. The target disk or object store has its own ceiling.

The backup runs at the lowest of these. Improving the link does nothing if the source disk is the limit.

What does the arithmetic look like on a real-sized example?

Example (illustrative): fs01 holds 8 TB (8 x 10^12 bytes) and is backed up across a 1 Gbit/s link to backup01. The window is 22:00 to 08:00, which is 10 hours. Assume 70% of the link is usable: 125 MB/s x 0.7 = 87.5 MB/s.

  • Time for a full: 8,000,000 MB / 87.5 MB/s = 91,429 s = 25.4 hours. That is more than twice the window.
  • Rate needed to fit 10 hours: 8,000,000 MB / 36,000 s = 222 MB/s, which is 222 x 8 = 1,778 Mbit/s, about 1.78 Gbit/s. The 1 Gbit/s link cannot do it even at 100%.
  • If the 8 TB were really 8 TiB, the size is 8 x 2^40 = 8,796,093,022,208 bytes, and the time rises to 8,796,093 MB / 87.5 MB/s = about 100,527 s = 27.9 hours.
  • Incremental: if 3% of the data changes in a day, that is 240 GB, or 240,000 MB. 240,000 MB / 87.5 MB/s = about 2,743 s = 45.7 minutes. It fits the window with a wide margin.
  • With a 10 Gbit/s link at the same 70%, 875 MB/s, a full takes 8,000,000 / 875 = 9,143 s = 2.5 hours, if the source and the store can both keep up.

The 70% share and the 3% change rate are assumptions chosen for the example. Replace both with measured figures.

When the answer does not fit, the arithmetic also shows the three levers: raise the rate of the slowest stage, reduce the data that must move in one night, or lengthen the window. Which lever is cheapest depends on which stage is slowest, which is why measuring each stage comes before buying anything.

Why do incremental backups exist?

One reason is that the arithmetic above gets worse as data grows and the window does not. Veeam describes periodic active fulls as "resource-intensive" and consuming "a considerable amount of network bandwidth", and offers synthetic fulls as an alternative (Synthetic Full Backup). Microsoft says a SQL Server differential backup "can be much faster than creating a full backup", because it "records only the data that has changed" (Differential backups (SQL Server)).

The same Microsoft page names the catch: a differential grows as time passes since its base, and can "approach the differential base in size", and restoring needs more steps. Incrementals shrink the nightly window and lengthen the restore path.

How can I measure my own rates?

  1. Measure the link. iperf3 is a tool for testing network throughput. Its documentation says -s runs "in server mode", and -c host runs "in client mode, connecting to the specified server". By default "a test consists of sending data from the client to the server", and the -R flag reverses the direction (Invoking iperf3). The test duration is set with -t in seconds, "default 10 secs". On the backup target run iperf3 -s; on the source run iperf3 -c backup01 -t 30. Run it again with -R to test the direction used by a restore.
  2. Read the other stages from the backup job itself, if the product reports them. You want the data read, the data transferred and the elapsed time. Divide each by the time to get a rate for that stage. If the transferred bytes are far fewer than the bytes read, the processing stage is shrinking the data.
  3. Time a real first full on a representative subset, for example 5% of the data, and scale up. Keep the units the same at every step.
  4. Convert, then compare with the window. Rate in MB/s = Mbit/s / 8. Time = size / rate.
  5. Ask a provider for the measured rate of the last full, with the size and the elapsed time, not just the link speed on the contract.

iperf3 measures the network path only. It says nothing about the source disk or the target.

Risks and limits

  • Risk: an iperf3 test fills the link for the test duration. Run it outside business hours, because it can slow other traffic and a running backup. There is nothing to roll back; it stops when the time ends or you cancel it.
  • These commands have not been run or lab-tested.
  • The arithmetic assumes a steady rate for the whole copy. Real rates change as the backup passes busy areas of the disk, other jobs and other users.
  • The table gives line-rate best cases. Every figure in it is a floor.
  • Compression and de-duplication can lower the bytes sent. The ratio depends on the data, and no generic figure is claimed here.
  • The example uses a made-up server, a made-up link share and a made-up change rate.
  • The first full over a slow link can take days; later runs are faster only if the product sends changes alone.

NetLeverage includes a daily full disaster-recovery backup of each hosted server at no extra charge. Talk to an engineer.

Sources