alt Hacker NewsWhat‘s the advantage of using ConnectX-5 Ex VPI NICs instead of much cheaper ConnectX-3 VPI NICs to connect two machines directly, other than PCIe 4.0 instead of PCIe 3.0? Can they offload more tasks when doing RDMA? Solid information is hard to come by.
Replies
olavgg • today at 10:34 AM
Yes CX5 can offload more. I believe CX4 has similar offloading capabilities as CX3, except that it supports 100G.
Another note: In my experience, RoCE works much better on CX4+ generation. CX3 is best with Infiniband. I think some firmwares on the CX3 generation, has a messed up config for RoCE. But running Infiniband is not a complex task, is way easier than people think, like 10x easier and faster to setup than Ethernet.
The machine only has pcie4x4 so 50Gb bandwidth, pcie3 would halve that to 25Gb
Thats the problem with these AMD laptop class cores, they have very little IO. They have been saying they will release in a desktop form factor, but then it probably wont have such good memory bandwidth...
The Nvidia boxes have 200Gb ethernet thats much more useful for clustering.