hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.
any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.
alt Hacker News
any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.