Ceph for private cloud: replication against erasure coding, latency and recovery load

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

For virtual machine disks, start with three-way replication: size=3, min_size=2, failure domain host. Choose erasure coding only when capacity matters more than small-write latency, you have at least seven hosts for a 4+2 profile, and the cluster is on Tentacle or later so the newer erasure-coding write path is available. There is no in-place conversion: an existing pool means a new pool and moving every object. Whichever you choose, size the cluster for recovery, because that is where the two designs differ most and where an undersized cluster hurts its tenants.

Interpretations are editorial; no lab testing was performed.

Why it matters

The usual comparison stops at capacity. Ceph's own figures put a 4+2 profile at 1.5 times the stored data against 3.0 for size=3, so erasure coding appears to halve the disk bill. The same page then warns: "Do not mistake erasure coding for a free lunch: there is a significant performance tradeoff, especially when using HDDs and when performing cluster recovery or backfill."

In a private cloud the cost lands on tenants who did not choose it. The decision therefore needs three answers: what a small write costs, what happens to input and output (I/O) when failure domains are lost, and how much read and network load a rebuild creates.

Terms used below: an OSD is the daemon that owns one storage device. A placement group (PG) is the unit of data Ceph places and recovers. A failure domain is the thing CRUSH, Ceph's placement algorithm, keeps copies apart across, usually a host. In k+m, k is the number of data chunks and m the number of coding chunks.

How it works

The replicated write path

The architecture guide describes it directly. The client writes to the PG's primary OSD. The primary sends the object to the secondary OSDs, confirms they stored it, and only then reports success to the client.

With size=3, a client write therefore waits for the primary's own write and for the slower of two network round trips with a device write at the far end. So latency is set by the slowest of three device writes plus the network hops to reach two of them (our reasoning from the write path, not a published figure).

The erasure-coded write path

In an erasure-coded pool the primary also receives all writes. It encodes the payload into k+m chunks and sends each chunk to its OSD. The guide describes the write as finished when the chunks are acknowledged on each OSD in the acting set.

That gives two penalties for block storage:

• More devices in the fan-out, for a full-stripe write. A full-stripe 4+2 write waits on six devices rather than three, so any one slow drive sets the latency. This is inference from the write path the guide describes, not a published figure. We found no page saying how many shards a FastEC partial write touches, and that is the common write on a virtual disk, so measure it with the test below.
• Partial writes. A virtual disk mostly issues small overwrites. By default erasure-coded pools support only full-object writes. Block use needs allow_ec_overwrites, which requires BlueStore and which the Umbrella edition says has "no downside" and is best practice to enable routinely.

What Tentacle changed

Since Tentacle, a per-pool setting, allow_ec_optimizations, improves performance for smaller I/Os and eliminates padding, which reduces wasted space on small objects. Proxmox's Ceph chapter calls this path FastEC, the name used below, and says it adds partial reads and partial writes. Ceph's development edition goes further: small reads come directly from the relevant shards, and small writes "no longer require reading and rewriting the full stripe". The Tentacle and Umbrella pages do not carry that sentence.

The limits matter more than the headline:

A larger stripe unit improves small reads but slightly reduces small sequential writes. With FastEC, raising k costs little, but m still hurts writes, and Ceph recommends m no larger than 3 for block and file workloads.

The metadata rule is unchanged. RBD and CephFS must keep metadata in a replicated pool, so an erasure-coded virtual disk always depends on two pools.

Reads, healthy and degraded

A healthy replicated read needs one OSD. An erasure-coded read needs enough chunks to decode, which is k, unless FastEC can satisfy a small read from the shards holding that range. With a shard missing, the data must be reconstructed from the survivors, so a degraded read touches several OSDs.

What each design does when failure domains are lost

Replicated pools can lose one fewer OSD than the replica count without data loss. Erasure-coded pools tolerate m losses. Availability is a separate matter and depends on min_size, the number of copies or shards below which a PG refuses I/O. The same page says the number of OSDs that can fail without data becoming unavailable "is usually two". The table below follows the min_size rule instead, and we read that sentence as loose, so validate the two-host failure case in an isolated test design.

The table assumes host failure domains and worst-case placement, and it is derived from the rules above. A PG in the peered state has too few copies to serve clients. Recovery can still run there, and from Octopus onwards an erasure-coded PG can recover with k shards present even if min_size is higher.

Ceph's erasure code page recommends min_size of k+1 or greater. Proxmox VE's Ceph chapter says Ceph recommends k+2, which for 4+2 means all six shards, so any single failure would stop I/O.

Keep min_size above 1. Ceph says size=2 or min_size=1 in production risks data loss and belongs only in emergencies, temporarily.

Where recovery load comes from

When an OSD goes down, it stays in until the Monitors mark it out, after 600 seconds by default unless noout is set. Then CRUSH remaps its PGs and data moves.

Ceph uses two mechanisms. Log-based recovery replays recent changes from the PG log and suits a short outage. Backfill scans and synchronises a PG's whole contents and is used when the log cannot drive recovery, such as for a new OSD in the PG.

The read cost differs sharply. Ceph's CLAY documentation states that with Jerasure or ISA-L, repairing a failed OSD holding S of data reads k × S. A replicated pool reads S, copied from a surviving replica.

The 20 minutes per TiB per 10 Gb/s is Ceph's own planning figure. The rest is arithmetic for illustration. Real recovery spreads across many links and is usually set by disks and the scheduler, so these are volumes to reason with, not predicted durations. A failed host multiplies them by its OSD count.

CLAY exists to cut that read, but it is deprecated since Umbrella and due for removal in the Vampire release, and Proxmox's table shows FastEC does not support it. The Tentacle page has no such notice. Do not plan a new cluster around it.

osd_max_backfills defaults to 1 and caps the backfills to or from a single OSD. The development edition adds that, because an erasure-coded PG must reserve all its shards' OSDs at once, wide erasure-coded pools are frequently limited by this "even when the disks are near-idle". The Tentacle and Umbrella pages do not say so. If it holds on your release, a wide profile can recover slowly on an idle cluster and extend the time spent with reduced redundancy, so watch for it in your own recovery test.

How the scheduler shares the disk

With the mClock scheduler, each profile reserves a share of the OSD's measured IOPS capacity.

Ceph's notes place backfill, scrub, snap trim and PG deletion in the background best-effort class, which has only a 5% reservation in each profile, so the recovery column does not describe a full rebuild by backfill. The profile percentages apply to a capacity figure measured at OSD start-up by OSD bench. Implausible results above a threshold (500 IOPS for hard disks and 80,000 for SSDs) are rejected in favour of the default value, so a wrong capacity figure skews every percentage. Check it with the option that matches the device type:

ceph config show osd.0 osd_mclock_max_capacity_iops_ssd
ceph config show osd.0 osd_mclock_max_capacity_iops_hdd

Under mClock, osd_max_backfills and the osd_recovery_max_active family are held at their defaults and need osd_mclock_override_recovery_settings to change. Ceph recommends leaving them alone because the profiles are tuned around them. Change the profile first, and treat it as a live change to how every OSD shares its disk. high_client_ops slows recovery, which lengthens the time the cluster runs with reduced redundancy, so note the current profile and put it back afterwards:

ceph config get osd osd_mclock_profile

Treat a change to the mClock profile as a separate performance experiment. Record the installed release, current profile and workload baseline; compare client latency and recovery time before selecting a new profile.

The network during recovery

Ceph asks for at least 10 Gb/s, with 25 Gb/s for substantial workloads and notes dense nodes often warrant 100 Gb/s. Proxmox warns that a single NVMe OSD can saturate 10 Gbps, and that recovery traffic can disturb corosync, its cluster communication layer. Corosync belongs on its own link.

A separate cluster network is optional. The released Tentacle troubleshooting page describes the trap: if a private network or a host link fails while the public network works, OSDs report each other down and flapping follows. On two-port nodes it suggests considering both links for a public network only. The development edition's troubleshooting page also says every host and switch port must use the same MTU, because mismatches pass small heartbeats and drop large packets. The Tentacle and Umbrella pages have no such passage.

What to choose and why

The choices below are our recommendations, based on the rules above.

Ceph also advises against k above 4 or m above 2 until the consequences are understood. 6+3 exceeds both limits. Ceph's stated offset is that FastEC makes a higher k "more viable", with m no larger than 3 for block and file workloads. It says most deployments need at least k+m failure domains and notes operational advantages at k+m+1. Below that, a lost host leaves nowhere to rebuild the missing shard, so PGs stay degraded until it returns. The same applies to size=3 on exactly three hosts (our inference from the failure-domain rule).

Reserve free space for the failure, not the steady state

The fullness ratios default to 0.85 nearfull, 0.90 backfillfull and 0.95 full. Above backfillfull, backfill waits. Ceph's hardware guide warns that if a host holds a large share of capacity and fails, recovery can push OSDs past the full ratio and halt operations.

Worked example, invented for arithmetic: seven hosts with 20 TiB raw each, 140 TiB in total. After one host fails, 120 TiB remains. Staying under nearfull means no more than 102 TiB of raw space used, or about 73% of the original raw capacity.

This assumes even data distribution, so verify with ceph osd df.

Hardware that changes the answer

• Drives. Ceph says enterprise SSDs with power loss protection are best.
• Hard disks. Ceph notes the very slow recovery of a hard-disk cluster means a lengthy period of increased risk. If hard disks are required, placing WAL and DB on SSD can significantly improve write latency.
• Memory. Ceph recommends server RAM above OSD count × osd_memory_target × 2, partly because recovery raises consumption.

Creating the pools

Replicated, with the Ceph command:

A replicated-pool design would specify three copies, min_size two, host-level failure separation and RBD initialisation. Have the storage engineer check the CRUSH rule and capacity before translating that specification into a creation procedure.

Erasure coded, with the options fixed at creation set first. Confirm every Monitor and OSD is on Tentacle or later before the final set command, because it cannot be undone:

An EC design specification would record k=4, m=2, the supported ISA/Reed-Solomon combination, host failure domains and the chosen stripe unit. RBD requires the appropriate overwrite support. Enabling EC optimisations is a separate irreversible compatibility decision, so this article supplies no activation command; validate the complete release and rollback requirements on a disposable pool first.

An image then needs the replicated pool for metadata and the erasure-coded pool for data:

The RBD design must identify both its replicated metadata pool and EC data pool. Confirm that the intended image actually uses that data pool before interpreting a benchmark.

The profile cannot be changed later: a different k, m or plugin means a new pool and moving every object.

Measure it on your own hardware

We found no current Ceph or Proxmox document that gives a latency ratio between these designs, and we are not offering one. Test on the hardware you will run, on a test cluster or with test pools and images only.

For an isolated benchmark, specify disposable images, 4 KiB random writes, queue depth, total bytes and identical client resources. Write tests overwrite their targets. Record tail latency and recovery completion as well as throughput, and ensure the test plan cannot select a production image.

rbd bench exercises the partial-write path that virtual disks use. rados bench writes whole objects, defaulting to 4 MB, so it does not represent small overwrites.

Then repeat the test while the cluster is recovering, because the result you care about is the worst one. On a test cluster only, mark an OSD out:

A failure experiment must be planned on an isolated cluster with known object placement and recoverable test data. Specify the failed host or OSD, expected PG states, stopping limits and restoration sequence. Changing membership starts data movement and is not a diagnostic command to run on a busy production cluster.

During that run, read:

Risks and limits

• Planned maintenance. ceph osd set noout stops downed OSDs being marked out. It also stops automatic recovery from a genuine failure while set, and raises a health flag. Run ceph osd unset noout when finished.
• Irreversible choices. Profile, stripe unit and FastEC cannot be reversed in place. Decide them on a test pool first.
• latest documentation. Several pages were read at /en/latest/, which carries a development-version notice. The erasure-coding rules were checked against the Tentacle and Umbrella editions, except the statements named above as development-edition only: the full-stripe sentence, the near-idle warning, the MTU rule and 6+3. The near-idle warning and MTU rule may well apply to released versions, but the released pages do not say so. Confirm any option name on your release with ceph config help <option>.

Sources

Opened 2026-11-10.

• Ceph: Erasure code (Tentacle)
• Ceph: Erasure code (latest)
• Ceph: Erasure code profiles
• Ceph: CLAY code plugin
• Ceph: Architecture
• Ceph: Pools
• Ceph: Placement group states
• Ceph: Monitoring OSDs and PGs
• Ceph: mClock configuration reference
• Ceph: OSD configuration reference
• Ceph: Monitor configuration reference
• Ceph: Network configuration reference
• Ceph: Hardware recommendations
• Ceph: Health checks
• Ceph: Troubleshooting OSDs
• Ceph: Control commands
• Ceph: ceph(8), rados(8) and rbd(8)
• Proxmox VE: Deploy Hyper-Converged Ceph Cluster

This article was originally published on 10 November 2026.