I was looking for something more general-purpose, such as a decommissioned quad-socket x86 with 80 or more AVX512 cores and a terabyte of RAM, which I suspect will be a strong limiting factor. The possibility of adding some older Nvidia accelerators is interesting as well even though they won’t be able to hold the whole model in HBM and will need to stream it over PCIe 4, which sucks.
That way, when not running inference, the machine can easily host VMs for other experiments and general housekeeping functions.
alt Hacker News
I was looking for something more general-purpose, such as a decommissioned quad-socket x86 with 80 or more AVX512 cores and a terabyte of RAM, which I suspect will be a strong limiting factor. The possibility of adding some older Nvidia accelerators is interesting as well even though they won’t be able to hold the whole model in HBM and will need to stream it over PCIe 4, which sucks.
That way, when not running inference, the machine can easily host VMs for other experiments and general housekeeping functions.