InfiniBand

InfiniBand is an open-standard, high-speed, low-latency interconnect technology used to network servers, storage, and accelerators in high-performance computing and large AI clusters. Unlike conventional Ethernet, it uses a switched-fabric architecture with credit-based, lossless flow control and offloads most of the communication work from the CPU into the network adapter, keeping latency in the microsecond range even when thousands of nodes exchange data constantly. It is the dominant fabric stitching GPU servers together in the largest training systems — the plumbing that makes a warehouse of accelerators behave like one machine.

How it differs from ordinary networking

InfiniBand's defining feature is native Remote Direct Memory Access (RDMA): one node can read or write another node's memory without involving either CPU in the data path, eliminating the kernel networking stack, its copies, and its latency from every transfer. The fabric carries clustering, storage, and management traffic over a single link, and lossless delivery is guaranteed at the link layer rather than patched over by retransmission. Speeds are described in generational tiers; current NDR-generation links reach 400 Gbps per port at the standard 4x width, with the follow-on generation doubling that again. In GPU clusters it pairs with GPUDirect RDMA, letting data move directly between GPU memories across nodes without a detour through system RAM — and the collective operations that dominate distributed training, the all-reduces issued by NCCL, are what actually ride the fabric.

Why the fabric is the bottleneck that matters

Distributed training techniques such as tensor parallelism and data parallelism synchronize constantly; every optimizer step waits on gradient exchange. Past a certain cluster size, the interconnect — not the accelerators — sets the training speed, which is why frontier clusters spend enormous sums on the network and why "how many GPUs" is a less meaningful number than "how well are they connected." Ethernet-based alternatives running RDMA over Converged Ethernet (RoCE) have closed much of the gap and power some very large clusters, but InfiniBand remains the reference architecture the alternatives measure themselves against.

Why it appears in sovereignty discussions

InfiniBand matters to the sovereign-tech conversation precisely because you will never have it at home — and do not need it. The fact that frontier-scale training depends on specialized, expensive, centrally procured fabric is a structural reason why training the largest models remains concentrated in a handful of facilities, while inference on open-weight models needs none of it: a single machine with adequate memory bandwidth serves a capable model with zero fabric involved. Understanding InfiniBand clarifies exactly where the centralization pressure in AI lives — in training economics, not in running models — and therefore where sovereignty is winnable today. Bitcoin mining offers the instructive contrast: hashing is embarrassingly parallel, each machine needing only a thin Stratum connection to a pool, no shared memory, no synchronized state. That is why hashrate decentralizes across garages and home racks while model training does not. The lesson for a home-lab builder: put the money where your workload's real bottleneck is — for local inference that means memory capacity and bandwidth on one box, not networking between boxes.

There is also a repair-economics parallel a miner will recognize: fabric hardware — switches, adapters, optics — depreciates on the same brutal generational curve as ASICs, and decommissioned datacenter interconnect floods the used market just as retired hashboards do. Home-lab builders sometimes pick up previous-generation RDMA gear cheaply to link a pair of inference or storage boxes; it works, but the watts, noise, and driver overhead usually argue for the simpler design of one well-provisioned machine. The larger point stands either way: know your workload's communication pattern before spending on the network layer — for embarrassingly parallel work like hashing, the correct interconnect budget is approximately zero.