Or, just maybe, they share a key property with the mounds of popular writing, journalism and random internet accretions they were trained on: they seem total rubbish on anything you know about deeply, but brilliant on everything else.

Their math is as just as good as their rambling writing and their Ghiblified poster art. Most people don't seem to realize that.

Writing is a very slippery thing. “Good” writing from 200 years ago reads as unbelievably dense and overwrought to most people today. So even if you made an LLM that was “good” at writing, how long would it stay that way? AI systems can put out more writing per day than all of humanity combined, meaning it can also generate and beat to death cliches much more efficiently.

> Today, the mathematicians are weeping, because the machine smashed their tower first.

I read this passage and thought, "has Derek even talked to software developers"?

That feels to me like the first discipline to go through this, where their daily activities are massively changed by GenAI.

Language models don't have a characteristic style you can spot from a mile away. They haven't had it since before ChatGPT.

What is true, is that chat models (and by extension agents) have a very characteristic style after their LLM went through post-training. But the pre-trained language model doesn't have that style.

So if you want to write well with an LLM, you could and it is a well understood problem. It is more a matter of that not being of much interest, as it competes with having the model be good for chat interfaces.

How to have a well-aligned, RL'ed, safe or thinking model, which also writes well in long-form, is a bit an open problem. Though presumably agents could do it when they have access to a pretrained model.

You can just tell them to adopt a different style. To make sure, add another pass with an agent that flags and/or fixes LLM tics. Humans typically do several passes over writings that they intend to publish, why are we expecting agents to do it in one shot? Just like the superpowers skill for software development does a spec phase, planning phase, several review rounds, etc., if you do the same with writing you get very passable results.

Makes me wonder whether it's a fundamental limitation of the technology or more like missing a good set of tasks in the training pipeline

Gemini's (at least for Flash 3.6 - 3.8) writing feels quite "human" to me

Math and programming languages are formal. Meaning they don't have any ambiguity in them. Every statement is what it is. That means they are very rigid and there's not a lot of variance. Natural languages are more diffuse and a written piece is more the reflection of the author than anything else.

That is why it's so disturbing. After several paragraphs, you realize there's nothing human there even when the actual information is present. People that do create AI piece are OK with it because they already have the context, but for the reader, it's always the uncanny valley.

There's no money in it. There's money in programming, and math is a good target to give the models superhuman reasoning abilities, but it already writes well enough for middle managers to write all of their emails.

>Isn't it extraordinary that these models still can't write well?

It's not at all extraordinary if you understand how they're trained. For math and programming it's possible to generate an infinite amount of training data with correct answers, as compilers and theorem provers can guarantee correctness. For writing, the quality is not objective, it's determined by human preference, so you need humans in the loop, which slows things down massively and limits the amount of data that can be generated.

Recently a Canadian immigrant won a bunch of the highest French literature prizes for his novel, which then turned out to be an LLM fabrication.