logoalt Hacker News

trouve_searchtoday at 7:18 PM1 replyview on HN

Yes, it runs in vllm happily. It gets >900TPS output reliably on a single 5090 with the nvfp4 model.

It's clearly worse than vanilla 26B-A4B, and lacks some things like structured outputs, and gets some tool calls wrong.

So you have to find a usecase or a hand rolled harness that leverages the cerebras-level TPS while not going off track during (even short) tasks.


Replies

npodbielskitoday at 8:35 PM

And it fails on rocm of course. This engine is such a hassle on AMD.

show 1 reply