alt Hacker NewsDoes it though? Let's say you use an LLM to write code, do you think it benefits from the fact that the same model is capable of translating corporate messaging into internet slang for marketing?
I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.
Replies
Or would it be worse as it wouldn’t have the broader knowledge to understand the reason you’re coding what you are anyway, understand the logs and iterate?
Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.
Isn't this just the Bitter Lesson over and over again? Specialised approaches get outperformed by more compute and a simpler general approach pretty much every time.
They need some of that fluff to keep up the illusion of intelligence and sycophancy that gets people to rely on them. Additionally, they need to make sure they don't prune so much that the model size seems manageable on consumer or standard business hardware, otherwise more people may become tempted to self-host a distilled or other semi-open model. The model size is also a marketing point.