RCDERKR Certified DataCenter Expert

Expert fabric design starts from the traffic, not the topology. This module dissects distributed training at the wire level: data/tensor/pipeline/expert parallelism and the collectives each one generates (AllReduce, AllGather, ReduceScatter, All-to-All), then builds the quantitative toolkit — ring-AllReduce data volume 2S(N-1)/N, algorithm-vs-bus bandwidth, halving-doubling and tree variants, and how NCCL selects algorithm and protocol (Simple/LL/LL128) per message size. You finish able to convert a training-job spec into a fabric bandwidth and latency requirement with defensible arithmetic.

The reference architecture for modern GPU backends is the rail-optimized Clos: each of the 8 NICs on an HGX-class node wired to its own rail leaf so intra-rail collectives complete in one hop. This module covers the full design discipline — rail-to-leaf maps, 1:1 non-blocking spine sizing, radix math on 51.2T ASICs (Tomahawk 5, Jericho3-AI, Spectrum-4 class), 400G/800G OSFP/QSFP-DD optics and DAC/AOC reach budgets, multi-plane growth from 256 to 32k GPUs, and where rail-optimized breaks down (inter-rail traffic, MoE All-to-All) versus a conventional fat-tree.

RoCEv2 moves RDMA verbs over UDP/4791 and assumes the fabric will not drop — a promise Ethernet only keeps if you engineer it. This module goes to the ASIC: shared-buffer architecture and per-priority buffer carving, DSCP-to-TC-to-priority-group mapping, PFC (802.1Qbb) frame mechanics, headroom calculation from cable propagation delay, link rate, and MTU (2 x flight-time + worst-case in-flight frames), PFC watchdogs, and the failure modes — pause storms, head-of-line blocking, and cyclic-buffer-dependency deadlock in Clos fabrics.

PFC is the safety net; congestion control is the strategy. This module masters DCQCN end to end: WRED-ECN marking curves (Kmin/Kmax/Pmax) on the switch, CNP generation and reaction at the NIC, the alpha rate-reduction/recovery state machine, and the timer/byte-counter increase phases — then how to tune all of it for incast-heavy collective traffic so ECN does the work and PFC almost never fires. Covers interaction between marking thresholds and buffer occupancy, per-queue vs per-port marking, and the emerging alternatives (HPCC-style INT-based control, receiver-driven credit schemes) you must evaluate as an expert.

The defining architecture decision of the decade: NDR/XDR InfiniBand with its subnet manager, credit-based flow control, adaptive routing, and SHARP in-network reduction — versus Ethernet fabrics (standards-based RoCEv2, NVIDIA Spectrum-X adaptive-routing/telemetry enhancements) and the Ultra Ethernet Consortium's UET transport with packet spraying, out-of-order delivery, and modern sender/receiver congestion control. This module builds the evidence-based decision framework: performance per collective type, operability, multi-vendor leverage, failure behaviour, and five-year TCO, so your recommendation survives both the CFO and the ML platform team.

A GPU cluster is three networks: the backend compute fabric, the storage fabric feeding checkpoints and datasets (NVMe-oF/RDMA to parallel filesystems like Lustre/GPFS-class or object stores), and the frontend/management network carrying orchestration, in-band telemetry, and tenant access. This module designs all three coherently: EVPN-VXLAN (RFC 8365) multi-tenancy with symmetric IRB on the frontend, lossless-class treatment for storage RDMA without starving compute collectives, DPU/SmartNIC-based tenant isolation, and how GPU-as-a-Service providers carve one physical cluster into secure tenant slices.

Expert fabrics are operated from evidence. This module builds the observability and acceptance discipline: gNMI/OpenConfig streaming telemetry into a Prometheus/Grafana stack, ASIC-level drop forensics (What-Just-Happened-class mirroring, mirror-on-drop), the counter hierarchy that matters (per-PG buffer watermarks, ECN marks, CNPs, PFC pause duration, out-of-sequence NAKs), and turning perftest and nccl-tests into automated acceptance gates — the pass/fail wall between 'cabled' and 'production'. Includes intent-based fabric validation (batfish-style pre-checks, automated LLDP/rail audits) and change-management for live training estates where a 3-week job must not die for a spine upgrade.

Most RCDE holders will not build greenfield — they will transform a live estate. This module covers brownfield-to-AI-fabric migration (parallel-fabric builds, workload cohort cutover, rollback criteria, cable-plant reuse analysis), capacity roadmapping where fabric, power, and cooling are co-designed (kW per rack from 15 to 130+, liquid-cooling implications for switch placement), and multi-site scale: cross-DC training feasibility, bandwidth-delay-product limits on synchronous collectives, checkpoint-replication design over DCI/DWDM, and hierarchical/asynchronous training topologies when the speed of light says no.

The finishing module is a compressed rehearsal of the RCDE Practical. Working from a realistic Indian GPU-cloud customer brief (capex ceiling, tenancy model, growth trajectory), you produce the complete design package — traffic model, rail-optimized topology, BOM, QoS/buffer plan, telemetry and acceptance strategy, migration plan — then build and tune the fabric on the RKR lab pod, survive a timed break/fix gauntlet, and defend the whole package orally before an RKR expert panel. Graded on the same 100-point rubric as the certification lab, so exam day holds no surprises.