alt Hacker NewsVanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
Just run /goal to optimise it and you should be good in less than an hour. Also best to use models that support speculative decoding.