logoalt Hacker News

marcosdumaylast Monday at 11:24 PM0 repliesview on HN

The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.