Not even a cluster of Mac Pros could run a dense 5T parameter model with RDMA, to my knowledge.
SOTA models are reportedly MoE, not dense.
alt Hacker News
SOTA models are reportedly MoE, not dense.