alt Hacker NewsThey also need to tell people what quantisation they're using. Because some 4 bit version is not the same as BF16, no matter what KLD suggests.
They also need to tell people what quantisation they're using. Because some 4 bit version is not the same as BF16, no matter what KLD suggests.
I'm using the unsloth dynamic Q4 and getting good results. I was running Q5, but Q4 gives more context headroom so I can run two agents in parallel with ~100k context each with 32GB vram.