how are you running the 2.4t model locally if you don't mind me asking
Oh I didn't; only a distilled 27B model ran locally. This was just to figure out if distilling caused the problem.
Oh I didn't; only a distilled 27B model ran locally. This was just to figure out if distilling caused the problem.