alt Hacker NewsTo the previous poster's point, soft distributions are useful, even saving the top 10 logits is significantly more training signal than just the final token.
To the previous poster's point, soft distributions are useful, even saving the top 10 logits is significantly more training signal than just the final token.