logoalt Hacker News

zozbot234 • today at 10:16 AM • 0 replies • view on HN

> Strata is not optimized to run the 3 [o]r 4-b[i]t versions, at least not yet.

How so? This is not obvious to me; AIUI Strata does support streaming weights for larger models via SSD. Of course the speed would drop quite a bit once you rely on SSD, but it ought to be workable. Since you're on a HEDT/server platform, you can also benefit by adding more SSD hardware to be accessed in parallel via the abundant PCIe lanes.