This theory doesn't hold up when you use base models (i.e., models with only pretraining, no RLHF or instruction tuning). They sound much more human. I've tested pairs (like Llama 3 base vs. instruct), and the base model can match human style much better than the instruct version can: https://www.pnas.org/doi/10.1073/pnas.2422455122

I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)

The other side of the same coin is that the core audience probably prefers such style, and that's why companies use it and AI now mimics it.

I cannot imagine a typical suburban American consumer tolerating a chatbot as blunt and direct as Linus.

Exactly. It’s been trained on petabytes of clickbait, engagement-farming content.

The slop-in, slop-out theory. I'm very much an advocate.

The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.

But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.

Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)

People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.

The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.

Symbolic AI is coming back, it's just that the mortar is random word association.

Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.