My guess is that tokenization is typically not (strictly) syllabic.

OP’s linked article mentions the importance for them of reading their writing aloud. Humans construct sentences with a (mostly unconscious) sonic influence. How things sound really matters.

My theory is that LLM prose sounds slop-like because the base unit for an LLM is not a single sound.

Thus, perhaps, you get the uncanny, meandering, hollow-feeling outputs, which feel vapid to human readers, without the reader really knowing quite why.

I think this is also part of the reason why LLMs are rewarded for and thus more strongly exhibit their faux-punchy constructions, such as the infamous “it’s not [X], it’s [Y].” The transformer model can, with current tokenization, articulate some features of “good writing”, but not quite all.

I think there are a number of fairly simple SOLNs to this (or ways to test this hypothesis) without making tokenizers purely syllabic across the board, which would probably be a mistake.

Whether prose output quality even matters for models which are not explicitly trained for creative writing is highly debatable, though.