logoalt Hacker News

adrian_byesterday at 5:05 PM0 repliesview on HN

When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".

In that file you can see all tokens and their corresponding numeric codes.