alt Hacker NewsIf Unsloth, DavidAU or mradermacher releases a quant that fits my hardware I'm definitely going to give this a shot.
Replies
sullivanmatt • yesterday at 5:49 PM
https://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF
I have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.
➕ show 1 reply
They were given day zero access to release their quantizations[0]. The lowest quant will still require 256GB VRAM to run inferences, but as always I'm amazed by the Unsloth team by how much they can reduce the requirements for these powerful LLM's, and the easy-to-follow guides to get started with using them.
[0] https://unsloth.ai/docs/models/mistral-large-4