Not sure what league people with a five-year old M1 Pro with 16GB of unified RAM fall into :)
Oh, that's a tough one. Do you have a fast drive? If you do you might try a 3 to 5B active MOE model, use the rest as cache to stream weights from NVMe. That would be your best bet. You'll be in the 'optimization league' ;)
alt Hacker News
Oh, that's a tough one. Do you have a fast drive? If you do you might try a 3 to 5B active MOE model, use the rest as cache to stream weights from NVMe. That would be your best bet. You'll be in the 'optimization league' ;)