RoCE v2 against InfiniBand for training clusters: PFC, ECN and congestion

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A training job built for RDMA does not move gradients through an ordinary socket. The adapter writes memory on the far node. InfiniBand was designed so that this transport sits on a link that does not drop frames for lack of buffer: the receiver hands out credits, and the sender transmits only while credit remains. RoCE v2 keeps that transport and puts it in a UDP datagram so it can cross an IP network. Ethernet does not hand out InfiniBand credits. Priority-based Flow Control (PFC) and Explicit Congestion Notification (ECN) are the two mechanisms operators add so the transport still sees a network that almost never drops.

An AI writing system produced this article for NetLeverage's editorial studio. A mark or a pause is a packet field or a frame, not a claim about subjective experience.

Nothing below was run in a lab. Commands are the ones printed in the cited documents. Writing PFC or ECN on a live training port can stall every flow on that priority. Read the counters first.

Why it matters

Collective operations in a training job are incast. Many GPUs send at once toward one peer, or toward one switch egress, for a short time. The buffer on that egress fills in microseconds. What the network does next decides whether the job slows evenly or whether an unrelated flow on the same link freezes.

InfiniBand answers that moment inside the link layer. RoCE v2 answers it with two different tools that must be set in the right order:

1. ECN asks the sender to slow down while the packet is still delivered.
2. PFC tells the previous hop to stop transmitting a whole priority, so the buffer does not overflow.

If pause is the only control, every flow on that priority stops, including flows that did not fill the queue. Zhu et al. showed that on a routed RoCEv2 fabric and introduced DCQCN so per-flow rate control would make pause rare. NVIDIA's RoCE page still says RoCE needs flow control, and that PFC on every endpoint and switch in the path is the normal way to provide it.

How it works

What the two fabrics share

The InfiniBand Trade Association describes InfiniBand as switched point-to-point links with remote memory operations that move data without software on the data path. Its 16 September 2014 RoCEv2 announcement says the specification keeps strict layering, adds IP routing, and is generated below the application API.

NVIDIA's MLNX_OFED RoCE page states the encapsulation directly. RoCE carries InfiniBand transport in one of two Ethernet forms:

The IANA registry, retrieved 10 November 2026, records roce on UDP 4791, "IP Routable RocE", InfiniBand Trade Association, 2014-10-17. TCP 4791 is reserved. Nearby names, including Ultra Ethernet Transport on UDP 4793, are different assignments.

The UDP source port is an opaque flow identifier so equal-cost multipath can spread datagrams without parsing the RDMA header. The IBTA integrators' list names Annex A16 as RoCE, Annex A17 as RoCEv2, Volume 1 as the protocol layer from Layer 3 up, and Volume 2 as Layers 1–2. The annex text was not opened, so this article does not claim byte fields from it.

InfiniBand's credit loop

Zhu et al. state that hop-by-hop, credit-based flow control prevents buffer-overflow drops. In the usual credit-flow-control model, a sender uses credits representing downstream buffer space; this explanatory sentence is background, not a quoted packet-level procedure from that paper. That is why the transport can stay simple. The paper does not include the credit-packet layout; that layout is in the architecture volumes, which were not opened.

A subnet manager assigns InfiniBand addresses and paths. On a RoCE port, NVIDIA shows the local identifier and the subnet-manager identifier as zero, and automatic path migration is unsupported. The association's overview describes per-hop and end-to-end CRC and a lossless I/O fabric, says NDR 400 Gbit/s is shipping, and places GDR 1.6 Tbit/s in a 2028 roadmap window. Its June 2023 TOP500 count is that page's citation, not a measurement for this article.

PFC and IP ECN do not create those credits. If link_layer reads InfiniBand, an Ethernet PFC recipe does not apply.

How a RoCEv2 queue pair chooses its packet

A RoCE port builds a GID table when an IP address appears on the Ethernet device. NVIDIA documents two entries for the same address, types v1 and v2. An IPv4 GID is an IPv4-mapped IPv6 address and, on RoCEv2, uses an IPv4 header. The GID index on a connected queue pair sets the mode. RDMA connection-manager applications expose the configured type through cma_roce_mode -d mlx5_0 -p 1. Changing that type affects new connections and belongs in an approved fabric-maintenance plan.

With RoCE enabled, UDP port 4791 is RoCE traffic. The roce_enable switch turns that off and drops the GID table. RoCE bytes are counted under /sys/class/infiniband/<device>/ports/<port>/counters/, not on the ordinary Ethernet device, because the offload bypasses that driver.

Priority is derived. For RDMA-CM the page gives SL = TOS >> 5, then PCP = SL & 7, and the service level affects VLAN priority only on tagged frames. Type of Service is 8 bits and DSCP is 6, so a DSCP value is multiplied by 4 before it is written as a traffic class. The page's example stores DSCP 24 as 96. If the job and the switch disagree on that shift, ECN and PFC land on a different queue from the one you configured.

PFC: pause, per priority, on a full-duplex link

NVIDIA describes PFC as the IEEE 802.1Qbb mechanism for pausing individual Ethernet priorities on a link. The IEEE abstract could not be independently reopened during this final review; no paid clause text or later standards-incorporation history is asserted here.

NVIDIA's Flow Control page (footer 29 May 2024) uses it for a split: lossless service for RoCE, best-effort for other Ethernet. There are eight priorities, 0 to 7. The page illustrates a mask selecting priority 4. Applying such a mask changes the adapter and can stall traffic; no executable configuration command is supplied here.

The same page shows a second path: LLDP and DCBX, so the switch can push PFC onto the host. A host set by hand and a host set from the switch will not match if only one path was finished. One end then pauses a priority the other never marked.

Pause is visible in ethtool -S only for priorities where PFC is enabled. The ConnectX-4 sample includes prio4_rx_pause, prio4_rx_pause_duration, prio4_tx_pause and prio4_tx_pause_duration. Their receive/transmit and duration interpretation follows the names; confirm units and definitions for the installed adapter/driver before comparing durations. A climbing transmit counter means this port is telling its neighbour to stop. A climbing receive counter means the neighbour is telling this port to stop. Poor throughput alongside rising pause counters makes priority stalls a candidate cause. It does not prove a frozen priority or exclude a GPU/application fault; correlate timing, rates and both ends of the link.

Pause-storm prevention on that page defaults to an 8 second stall timeout, or 100 milliseconds in auto mode. tx_pause_storm_warning_events counts a stall past the watermark. tx_pause_storm_error_events counts a stall past the timeout, after which the device stops transmitting pause. The lossless class is then no longer being enforced. Find the buffer or priority loop. Disabling PFC on one host while the switches still expect it does not repair that.

The RoCE page says global pause is possible and normally undesirable, because it stops every class. PFC on all eight priorities does the same. Keep training on one lossless priority. Leave management and telemetry on a priority that is allowed to drop.

PFC pauses a priority, not a flow. Zhu et al. record head-of-line blocking and unfairness on a Clos fabric: a paused link also stops traffic that only shared the link, and every flow in the class stops together. Eight classes cannot separate every collective. Quantized Congestion Notification does not cross IP, because its flow id was Layer 2 addresses and a router replaces the Ethernet header. DCQCN is the per-flow rate control they put on the adapter. It does not retire PFC: a flow starts at line rate when the host has no other active flow, and that burst can overflow a buffer before the first marks return.

ECN, then DCQCN

RFC 3168, September 2001, puts a two-bit field in the IP header:

A full queue still drops. The mark reports approaching congestion. It does not promise that memory remains.

NVIDIA's ECN page, for this RoCE v2 implementation, says those two bits occupy the IPv4 Type of Service field or IPv6 Traffic Class field and that every node on the path has to support ECN. The driver exposes two algorithms, per port and per priority:

• roce_np — notification point, the receiver side
• roce_rp — reaction point, the sender side

The documented enable, for a priority X from 0 to 7, is a write of 1 to:

/sys/class/net/<interface>/ecn/roce_np/enable/X
/sys/class/net/<interface>/ecn/roce_rp/enable/X

Reading those files is the check. Writing them changes live rate control. A coordinated deployment must verify the required notification and reaction settings at the correct endpoints. Parameters live under the matching params directory; read the names on the driver you have. The 2015 constants are not a current tuning sheet.

DCQCN splits the work three ways.

Congestion point (the switch). At an egress queue the switch ECN-marks a packet whose queue is past a threshold, using RED-style marking. Setting the two thresholds equal and the mark probability to 1 mimics DCTCP. The authors say that setting was not the optimum they later calculated.

Notification point (the receiving adapter). A CE mark means some queue was past its threshold. RoCEv2 defines a Congestion Notification Packet back to the sender. The state machine sends one immediately if the flow has not sent one in the last $N$ microseconds, then at most one every $N$ microseconds while marks continue. Their deployment used $N=50\mu s$. No marks means no notification.

Reaction point (the sending adapter). On a notification the sender stores the current rate as a target and cuts:

If no notification arrives for $K$ time units, alpha decays:

Their implementation used $K=55\mu s$, which must exceed the notification timer. Recovery uses a timer and a byte counter, not TCP slow start: fast recovery averages current and target for $F=5$ iterations, then additive increase steps the target. Notifications go out at high priority.

Mark before you pause, pause before you drop

ECN marking is an egress threshold. Pause is an ingress threshold. The paper's worst case is every egress queue drawing from one ingress, so pause must sit above the point where marking has started and below overflow. On their 32-port 40 Gbit/s switch, the upper bound on pause implied an ECN threshold below one MTU, which they call infeasible, so pause scaled with free shared buffer instead of a fixed value. Do not copy that kilobyte figure. Keep the order: mark, then pause, then drop. If pause fires first, DCQCN never sees the queue it was built to rate-limit.

What to choose and why

Our recommendation, not a vendor ranking: choose the fabric whose failure you can name. InfiniBand fails in the subnet manager, the credit stalls and the port state. RoCE v2 fails in a priority mismatch, a pause storm, or an ECN threshold that never marks.

The safe order when a job stalls

Start with read-only evidence. A counter alone does not authorise or establish the right configuration change. Names such as mlx5_0 and eth1 are the vendor examples. ibdev2netdev is the documented map from adapter port to Ethernet name.

1. Name the link layer. In the vendor's ibv_devinfo example a RoCE port still prints transport: InfiniBand. The distinguishing field is link_layer.

cat /sys/class/infiniband/mlx5_0/ports/1/link_layer
cat /sys/class/infiniband/mlx5_0/ports/1/state

InfiniBand means stop looking for PFC. The documented RoCE example also shows port_lid: 0 and sm_lid: 0. ofed_info -s prints the MLNX_OFED version so you follow the matching page.

2. Name the RoCE generation. Read gid_attrs/types/ and cma_roce_mode -d <dev> -p <port>. A v1 type on a job that was supposed to route is a configuration fault.

3. Confirm the marked priority. If the switch enables PFC on priority 3 and the queue pair emits priority 0, the lossless configuration is on a different queue. Fix the mapping before any buffer threshold.

4. Read both ECN enables for that priority, then the pause counters:

ethtool -S eth1 | grep pause

Rising pause duration with a quiet notification path suggests that ECN feedback is absent or ineffective; corroborate the receiver, sender and switch configuration. Rising tx_pause_storm_error_events means the adapter has stopped sending pause after the documented timeout.

5. Prepare any change only after diagnosis. The Flow Control page's switch session (dcb priority-flow-control, LLDP TLV selection) is one vendor syntax. Other switch systems use different commands for the same IEEE behaviour. A reviewed maintenance plan must coordinate host, switch and far host, with rollback and post-change verification. PFC is hop by hop. Do not aim the documented ibv_rc_pingpong traffic generator at a NIC inside a running training job.

This article covers the cited traditional PFC/ECN deployment model. Lossy or adaptive RoCE designs and Ultra Ethernet are outside its scope; the PFC requirement here is not a claim that every possible RDMA fabric uses this configuration.

Risks and limits

• Pause spreads. The 2015 experiments describe lower throughput and high variability when PFC is the only reaction, including flows that were not the congested ones. Eight priorities are not per-flow fairness.
• Line-rate start still needs a backstop. The paper says pause remains necessary because flows start fast. The RoCE page, footer 8 February 2026, still requires a form of flow control and calls PFC the normal method.
• The 2015 kilobyte bound is for the switch they calculated. Keep the order: mark, then pause, then drop. Their ConnectX-3 Pro note (about 10–20 congested flows at 40 Gbit/s) and ConnectX-4 note (over 200) are 2015 device statements, not a 2026 sizing rule. $N$ and $K$ belong to that deployment; read params/ on the host.
• Unopened text stays unclaimed. Credit-packet layout, InfiniBand's own congestion fields, and the paid 802.1Qbb clauses are not specified here. The abstract and NVIDIA's use of the standard are what was opened.
• No NetLeverage fabric, product or service level is described.

Sources

1. K. Ramakrishnan, S. Floyd, D. Black, RFC 3168, September 2001 — https://www.rfc-editor.org/rfc/rfc3168 (opened 10 November 2026). ECN codepoints and the full-queue drop rule.
2. IANA service-name registry — https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt (opened 10 November 2026). UDP 4791 roce, 2014-10-17.
3. IEEE 802.1Qbb-2011 abstract — https://standards.ieee.org/ieee/802.1Qbb/4361/. Access was blocked during the final review; operational PFC statements here rely on NVIDIA's Flow Control documentation instead.
4. IBTA, About InfiniBand — https://www.infinibandta.org/about-infiniband/ (opened 10 November 2026). RDMA data path, lossless-fabric wording, NDR and GDR sentences, June 2023 TOP500 citation.
5. IBTA RoCEv2 announcement, 16 September 2014 — https://infinibandta.org/infiniband-trade-association-releases-updated-specification-for-remote-direct-memory-access-over-converged-ethernet-roce/ (opened 10 November 2026). Layer 3 routing below the API.
6. IBTA Integrators' List PF32 — https://www.infinibandta.org/wp-content/uploads/2019/10/Integrators-List-PF32.pdf (opened 10 November 2026). Annex A16 and Annex A17 names only.
7. NVIDIA, RDMA over Converged Ethernet (RoCE), footer 8 February 2026 — https://docs.nvidia.com/networking/display/MLNXOFEDv582030LTS/RDMA+over+Converged+Ethernet+(RoCE) (opened 10 November 2026). Encapsulation, GID types, PFC requirement, counter path.
8. NVIDIA, Flow Control, footer 29 May 2024 — https://docs.nvidia.com/networking/display/mlnxofedv24104140lts/Flow+Control (opened 10 November 2026). PFC mask, pause counters, storm timeouts.
9. NVIDIA, Explicit Congestion Notification (ECN), footer 29 May 2024 — https://docs.nvidia.com/networking/display/mlnxofedv24104140lts/Explicit-Congestion-Notification-(ECN) (opened 10 November 2026). roce_rp and roce_np.
10. Zhu et al., Congestion Control for Large-Scale RDMA Deployments, SIGCOMM 2015 — https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p523.pdf (opened 10 November 2026). Credit-based link layer, DCQCN, and the statement that PFC remains required.

This article was originally published on 10 November 2026.