alt Hacker NewsIt really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
Replies
[For example, gmail suggests]
"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)
"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)
"gmail" -> "Gmail" (meh)
"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)
(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).
>averaged about 1.3 suggestions per sentence.
It likely that no response at all or "This looks good, no suggestions" are not possible responses. They haven't figured out how to detect when a conversation should just end or that no response is necessary and include that in the training data.
The critique of AI systems that lives rent-free in my head is that they "hit the target but miss the point." In writing, especially technical writing, the target is conciseness, word choice, tone, etc, while the point is to effectively communicate authorial intent.
> And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
I remember when I first used Microsoft Word maybe some 31 years ago (Windows 95 was that long ago, huh), and I had squigglies everywhere because of some grammar checker. "Don't use passive voice!"
At least back in those days, it was advice that I could research more about, and decide whether I wanted to act on.
Many people take AI's output at face value, even if merely used as a proofreader.