alt Hacker NewsI have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).
Replies
You're just going to have to try it. It won't take that long to quantize. Unless the server is currently serving traffic you should be able to get it going in under a day with a frontier model doing the driving for everything - quant, implementation, etc.
4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...
I have similar hardware and requirements. Spend the time to write a test process to try multiple quantization configs/multiple inference engine and throw at it a bunch of request you can replay and evaluate the results of. We got it down to a process and everytime there's a new model/options/inference engine version we just run the internal bench/eval suite and make a choice.
> we have restrictions that we can't use a chinese model!
Please expand on this if you can. Why can't you? Who mandated this?
One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.
4 bit quants can be fine, try it and see.