alt Hacker NewsXeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.