alt Hacker NewsThese are very good!
I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.
I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options.
So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8.
Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.