alt Hacker NewsHow are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal.
How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal.
No one is doing that for a model this size it would have to be so heavily quantized that it wouldn’t be useful - or you’d need to spend a half million dollars on hardware. People use hosted APIs. Open weight means cloud vendors can host it.