Cross-host GPU networking notes
Written on September 22nd , 2026 by cachemix
Notes from getting multi-node GPU jobs to actually use the fabric instead of quietly falling back to TCP. Writing this down because I lost most of a day to ordering problems that would have been obvious if I’d read the operator docs in the right sequence.
Two operators, and they are not interchangeable. GPU Operator handles the driver, container toolkit, device plugin, DCGM. Network Operator handles the NIC side: DOCA/OFED driver, the RDMA device plugins, Multus, and IPAM. Neither one does cross-host GPU traffic by itself.
Order matters. GPU Operator first, let the driver daemonset go Ready, then
Network Operator. nvidia-peermem is the module that lets the NIC DMA straight
into GPU memory, and it builds against the installed driver. If the driver
isn’t there yet you get a Network Operator that looks healthy and a
nvidia-peermem that silently isn’t loaded. Check with lsmod | grep
peermem on the node, not from inside a pod.
The default pod network is not the fabric. That was the part I kept
half-forgetting. Pods get eth0 on the CNI and that’s it. For RDMA you attach a
second interface via Multus — either SR-IOV VFs, or host-device if you’re
giving the whole NIC to one pod. NicClusterPolicy sets up the plugins;
HostDeviceNetwork / MacvlanNetwork define what gets attached.
Things that bit me:
- NCCL will happily run over the pod network and give you numbers that look
fine on two nodes and fall apart at eight. Set
NCCL_IB_HCAexplicitly and turn onNCCL_DEBUG=INFOat least once so you can see which transport it picked. If it says “via NET/Socket” you’re on TCP. - MTU has to match end to end. 9000 on the NIC, the switch, and the NetworkAttachmentDefinition. One of those at 1500 and you get throughput that is bad but not obviously broken.
- RoCE wants a lossless fabric. PFC and ECN on the switch side. This is not a Kubernetes problem and no amount of yaml will fix it.
- NUMA. The GPU and the NIC should be on the same socket. The device plugin won’t enforce this for you.
Validation loop I settled on: nccl-tests all_reduce_perf across two nodes,
compare against single-node. If cross-node is roughly in the same ballpark per
GPU, the fabric is being used. If it’s an order of magnitude worse, something
in the chain above is wrong and it’s almost always the second interface not
being attached the way I thought it was.
Still open: whether to use SR-IOV VFs or hand over whole NICs. VFs are more flexible and the density is better, but the config surface is larger and I’ve had VF allocation get stuck in ways that needed a node drain. Host-device is dumber and has not failed yet.