alt Hacker NewsThis is a serving bug or quantization issue. I had all kinds of issues that were like this on DGX Spark until I found a single-GB10 vLLM recipe [1] that uses Nvidia's NVFP4 quant. The community quants did not work well.
Another failure mode you may see is inordinately long CoT. Properly served, the model is good at calibrating its CoT length to the difficulty of the immediate task.
Replies
ryan-c • today at 6:09 AM
I get the same messages in a MLX Q8 quant - I assumed it was reacting to the harness, but it doesn't seem to cause issues.
➕ show 1 reply
I’m using Eugr’s recipe (TP=1) with Nvidia’s NVFP4 quant and I’m still seeing it sometimes! Rarely. But sometimes
Still works great though!