alt Hacker NewsIMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.
I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.
I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).
Replies
What date was the peak? If it is today it’s not something you can know so I assume you think the peak was many months ago.
Is there any reason to think that training weights has peaked rather than is accelerating? It feels like now they are increasingly able to pick some low hanging fruit by using the models in order to improve themselves and test optimizations.