alt Hacker NewsI think that this can be automated by using two LLMs: a stronger/more expensive for generating prompts and a weaker for actual classification. Approximate algorithm:
1. Give "strong" LLM the task formulation and some labeled examples. Ask it to generate a prompt for the "weak" LLM.
2. Run "weak" LLM on the training set with generated prompt from 1, use replies as features for a smaller ML model (logreg, decision tree etc).
3. Pick examples from the training set that your small model is most wrong about and ask "strong" LLM to generate one more prompt (like in 1), except this time you are using the misclassified examples instead of random.
4. Run "weak" LLM on generated prompt from 3, add results as one more feature for your model.
5. Repeat 2 - 4 until your token budget for this task is exhausted or required score on cross validation set is reached.
I was thinking about creating an open source library that implements this, but I'm not sure if anyone really needs it. I suspect that people who need something like this already made their own implementation.
I think there’s something to what you’re saying. “Systems I and II” theory about how our brain works feels similar and is why your ideas has some truth behind it or are otherwise naturally intuitive.
2 cooperating bodies thinking or acting as one moving force just makes sense, or so much more sense than a system that is one single approach but single minded, or even 2 regular LLMs working in tandem. It’s still not the same as them being fully cooperative or working hyper-cooperatively, and what you suggest technically forces that cooperation to some degree. It’s just ideal to have these aims view themselves as a “single body” like the system 1 and 2 concept works within our heads.
For this reason it’s why I feel the current loony approach where chatgpt mails Claude and refers to it by name or anything where 2 Standard (‘Selfish’) LLMs interact is generally ineffective or inefficient. They’re coerced to cooperate and naturally wouldn’t or have no natural imperative to do so.
They (‘Large’ LLMs atleast) have their own inherent preconceived views and idea of ‘self’ programmed in, but beyond this will naturally not always have the same ideas or shared perspective on what to do, which will naturally lead to issues within cooperation towards a shared goal. This would naturally impact the final product of the prompts or otherwise diminish returns on efforts made there. It’s all natural.