If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.

I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)