alt Hacker NewsThank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task.
Had a couple of questions:
1) You note that ARC-AGI is a meta-learning task, have you tried any meta-learning algorithms such as MAML?
2) Do you think this approach could extend to ARC-AGI 3? Or do you think the interactive environments require a higher level of complexity than what can be achieved with a small model?
Replies
I spent some time working with that approach of using LLMs to generate synthetic labeled data for use in training more specialized models. It mostly didn't work.
The problem was that getting the LLM to generate training data that sufficiently resembled real-world data was labor intensive and expensive. More labor intensive and expensive, it turns out, than just using real data.
What worked better was using the LLM to label the training data. But even there we had to be careful about introducing weird biases.
Glad to know you like it!
1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time!
2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)