Replication: How Much Bandwidth Do You Need to Catch Up After an Outage?

A replication connection must do more than carry new changes. After an interruption, it also needs to drain the backlog while production continues. A useful starting point is therefore: required payload throughput = ongoing change rate + backlog divided by the desired catch-up time. Port speed alone does not answer that question. What matters is the throughput replication can sustain between the actual endpoints.

Plan three operating phases separately

• Initial seeding: an initial data set must reach the second site. This has its own data volume and agreed time window.
• Normal operation: new changes must arrive within the permitted lag. Peaks during imports, maintenance or backups belong in the workload profile.
• Catch-up: after an interruption, accumulated and new changes compete for the same resources. A link that only just handles normal operation has no headroom for this.

The software determines what gets transferred. For snapshot-based SnapMirror replication, NetApp describes an initial baseline transfer followed by asynchronous snapshot updates. This is not a universal model for every replication system. Record both the logical change volume and the bytes actually transferred by your product. Compression, coalesced changes or a required fresh baseline can change those volumes.

Which measurements belong in the calculation?

Use a consistent data basis for change rate, backlog and throughput. A counter of uncompressed writes cannot be compared directly with compressed network bytes. In the model below, W is the volume of new data to replicate per second, B is the outstanding volume and C is the effective throughput available to replication. All three refer to the same payload data; protocol overhead and other traffic have already been accounted for in C.

• Measure the change rate across representative work cycles, not just overnight or as a daily average. Record the size and duration of peaks.
• Track backlog bytes separately from the age of the usable recovery point. A short transfer does not necessarily mean the data is current.
• Check achievable throughput with real replication and competing applications. Observe port counters, software limits and destination storage at the same time.
• Specify units explicitly: this article uses decimal MB and GB; 1 MB/s equals 8 Mbps. MiB/s and GiB require a different conversion.

The meaning of lag is also product-specific. NetApp describes SnapMirror lag using the snapshot timestamp and the time at the destination. It is not simply the duration of the last transfer. Check the definition of the counter you use and ensure clocks are synchronised before deriving an alert threshold from it.

Worked example: a two-hour outage, one hour to catch up

The following illustrative planning model is not a measurement or promise: assume a constant 25 MB/s of new payload data to transfer, two hours without transfers and then one hour to catch up completely. All changes remain available and must be transferred. There are no compression effects and no fresh full baseline is required. These assumptions must be checked for a real application.

1. 1Calculate the backlog: 25 MB/s × 7,200 s = 180,000 MB, or 180 GB.
2. 2Find the extra throughput needed to catch up: 180,000 MB ÷ 3,600 s = 50 MB/s.
3. 3Add ongoing production: 50 + 25 = 75 MB/s, equivalent to 600 Mbps of payload throughput.

Here, 600 Mbps is not a ready-to-order circuit size. Protocol overhead, concurrent traffic and operational headroom must also be accommodated. Their size depends on your setup. A blanket percentage cannot replace measurement. Conversely, a 1 Gbps port does not prove that replication will reach 75 MB/s.

A cross-check exposes the headroom: if the same replication only achieves C = 40 MB/s, subtracting the ongoing W = 25 MB/s leaves 15 MB/s to drain the backlog. 180,000 MB ÷ 15 MB/s gives 12,000 seconds, or three hours and twenty minutes. If C = W, the backlog does not shrink in this model; if C < W, it grows. For constant rates and C > W, the general relationship is: catch-up time = B ÷ (C − W).

Catch-up time is not RPO or RTO

The recovery point objective (RPO) describes tolerable data loss in time terms. The recovery time objective (RTO) describes the target time to restore service. Calculated catch-up time instead describes when a lagging copy becomes current under the model's assumptions. A one-hour catch-up target therefore means neither a one-hour RPO nor a service that is usable again within an hour.

Depending on the product, synchronous replication waits for acknowledgement from the destination and therefore responds to latency differently from asynchronous transfers. For example, Microsoft specifies average round-trip latency of no more than 5 ms and enough bandwidth for the write workload for synchronous Storage Replica. This is a product-specific requirement, not a universal L2 limit or a latency promise for a particular route. Higher throughput cannot remove a long acknowledgement delay.

When more bandwidth does not fix the backlog

First check whether the circuit is actually the limiting factor. NetApp, for example, documents global and per-relationship bandwidth limits that can interact. Read and write performance, processing and configured transfer schedules also belong on the checklist. More circuit capacity cannot override a lower software limit.

For TCP-based transfers, RFC 6349 identifies path MTU, round-trip time, bottleneck bandwidth and TCP windows among the relevant factors. A test using many parallel connections may show different throughput from the real application. Test replication itself as well as the transport. Increasing MTU on one endpoint alone is not reliable evidence of an improvement.

Acceptance testing: include recovery

1. 1Agree an authorised test window, load limits and stop conditions. Do not saturate a production connection without prior agreement.
2. 2Measure normal operation under representative load: document change rate, throughput, backlog and application health.
3. 3Create a defined backlog in an appropriate test environment. Then measure how quickly it actually shrinks while changes continue.
4. 4Check that log or snapshot retention is sufficient. If the software needs to reinitialise after a long interruption, the simple backlog calculation no longer applies.
5. 5Include the intended backup path and reverse direction. After a site switch, data may flow the other way; document a separate workload profile for that case.
6. 6Check the recovery point and a real application operation at the destination. An empty transfer buffer alone does not constitute a successful recovery test.

Include these details in a site connectivity enquiry

Provide both handoff locations, replication product and mode, measured change rates, planned initial seeding, tolerated interruption and desired catch-up time. Add concurrent services, encryption, MTU, RPO/RTO and backup-path requirements. This makes it possible to agree concrete capacity and acceptance criteria. Whether L2 is required or routed IP reachability is sufficient depends on the application; transport alone does not guarantee a recovery time.