It's because of "chunking" which human minds do as well. If you increase granularity, you increase permutations and required compute to match the same performance of existing systems.

GPT4's tokeniser for example made use of extended tokens for coding structures and with enough training on code was quite ahead of the rest for a while.

The tokeniser and its vocabulary makes a big difference.