alt Hacker NewsHF says activation function is "SiTU-GLU", but I can't find any info on that?
Anyone know / is this a typo?
Even the python code for inference seems to use normal activations.
HF says activation function is "SiTU-GLU", but I can't find any info on that?
Anyone know / is this a typo?
Even the python code for inference seems to use normal activations.
It stands for Sigmoid Tanh Unit Gated Linear Unit. Check the tech report/paper (page 7): https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_repo...