Nvidia DGX/HGX H100 — MRC Fabric Integration

NVLink for intra-node all-reduce · ConnectX-7 NICs for MRC inter-node fabric · 8 × 400G = 3.2 Tbps per node

DGX H100 — Node A H100 H100 H100 H100 H100 H100 H100 H100 NVLink 900 GB/s intra-node NVSwitch × 4 PCIe Gen5 — kernel-bypass RDMA CX-7 CX-7 CX-7 CX-7 CX-7 CX-7 CX-7 CX-7 8 × 400G = 3.2 Tbps → Plane P1–P8 leaf switches DGX H100 — Node B H100 H100 H100 H100 H100 H100 H100 H100 NVLink 900 GB/s intra-node NVSwitch × 4 PCIe Gen5 — kernel-bypass RDMA CX-7 CX-7 CX-7 CX-7 CX-7 CX-7 CX-7 CX-7 8 × 400G = 3.2 Tbps → Plane P1–P8 leaf switches MRC Fabric Planes P1–P8 · 7060XE7 Intra-node: NVLink / NVSwitch 900 GB/s per GPU · CPU uninvolved all-reduce stays on-node Inter-node: MRC Ethernet CX-7 MRC engine · zero-copy RDMA SRv6 uN SID encodes 32-bit EV · 128–256 EVs per QP vs InfiniBand HDR MRC: 8×400G = 3.2 Tbps per node IB HDR: 8×200G = 1.6 Tbps · 2× H100 GPU / NVSwitch ConnectX-7 NIC NVLink PCIe Gen5 400G fabric uplink NIC n of every node connects to plane n leaf switch — the wiring discipline that makes MRC work