alt Hacker NewsWaiting for llama.cpp support to land, but this might be a big deal for Strix Halo users.
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
Replies
hedgehog • yesterday at 7:22 PM
In my early testing it's way better both quality and speed on Strix Halo (posted recipe in sibling comment).
Using llama.cpp I one-shotted (2 hours) a reasonable asteroids clone on my strix halo/128 using the 1 bit quant, using my custom harness (which isn't anything exceptional).
It was tedious - a lot of second guessing itself, and quadruple checking things it fixed a couple of iterations back - but it got there and the result is a playable game.
Speed starts out strong, but definitely drops off as context grows. At the end (I think context about 70k) it was down to 12 output tps.
Mind a bit blown.