alt Hacker NewsNo, you can't divide the entire size by the expert count. A lot of weights are constant for all tokens, so total active count is ((2800-(shared)/896)*16 + (shared))
No, you can't divide the entire size by the expert count. A lot of weights are constant for all tokens, so total active count is ((2800-(shared)/896)*16 + (shared))
TIL, that makes a lot of sense, and thanks for the correction.