I think that would imply the creation of AGI (i.e. something as intelligent or more intelligent than mankind), which many consider to be science fiction at this point.

> bitter-lesson-pilled

The "bitter lesson" doesn't imply that AGI is coming, all it says is that letting AIs learn on their own yields better results than directly teaching them things.

They do improve, but the general creativity and sparkle we see with increasing scale comes mostly from scaling up pretraining/parameter-size, so it's quite slow and expensive compared to the speed (and decreasing cost) people have come to take for granted in math/coding in small cheap models. Hence the reaction to GPT-4.5: exactly as much better taste and discernment as it should have had based on scaling laws, yet regarded almost universally as a colossal failure. It was as unpopular as the original GPT-3 was when the paper was released, because people look at the log-esque gains from scaling up 10x or 100x and are disappointed. "Is that all?! What has the Bitter Lesson or scaling done for me lately?"

So, you can expect coding skills to continue to outpace the native LLM taste.

I think the author addresses this in saying that since AI output is statistically plausible by design its unlikely to improve in this area. Why do you think AI will get better in this way?

At least in part because some of Taste is fashion.

most (all?) models are fundamentally a regression toward the mean. Good taste is rarely, if ever, residing in the mean.

Regardless of how good the tools get, third-party tooling can never be a product differentiator unless you somehow manage to have exclusive access. Otherwise, everyone else out there can and will use the same tools you are. It's more a hedonic treadmill than a moat.