alt Hacker NewsI trained a small transformer in 1.5hrs and it beats many LLMs
Comments
I think you’re asking the right questions, sample inefficiency is horrible in modern LLMs. Despite this, I saw your analysis:
> The biggest increases in scores were due to
Modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm, etc.) More data diversity, better shuffling of data scaling up: 8 layers instead of 4
This is commonly called squeezing the lemon and is usually a bit of a last resort. You should be able to achieve near SoTa with your new method, before you squeeze any lemons. This is, because the old SoTa is typically not using new optimisers and thus your results will be distorted by a large margin.
In terms of sample efficiency I want to add two things:
Runtime per-puzzle fine tuning is a very good target that provides a LOT of information. People have not looked at evolutionary methods to harness induction since the 90ies - if I was to work on ARC ever again I’m fairly certain this is where I’d look.
Best of luck, padawan
I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop?
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
Sounds like a good day to be you, top 5 on Kaggle with a publication like this. It seems like you will be on a plane to SF shortly
Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
> I agree that its rare to see to face problem sets in real life where every problem is given at once. Even if it is (like an exam), humans can usually only attempt one at a time
Just one small snippet that I thought was interesting. I would always read through ~the entire exam before starting. Both so that I could find the problems most approachable to me, but also because sometimes it helps me figure out the rest of the questions :-)
> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is possible Guarantees no synthetic data can be used It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created)
I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
> Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised too
I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training.
I saw this on the community note for the last blog you wrote - anything to do here.
I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.
It means the questions including their answers are dependent. Ie, theres a data generating process for them, that the model uncovers. Like a KNN is known to have near Bayes accuracy as k/n to 0, n to infty, k to infty. The data reveals the dgp.
I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.
Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.
I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him?
So this achieves 44%, I'm wondering: To consider yourself a reasonably competent human, what % to achieve on ARC-AGI-1? Thanks in advance to anyone that knows, can answer!
Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?
Thanks for motivating me to work a bit on non-LLM things again :)
Just wanted to say that the rhabdo part mentioned on your site was really impressive! Speaking as a medical doctor and full stack engineer myself.
Nice. It took me an hour just to understand what this competition is about :)
Nice to see arc-agi-1 framed this way — I'd been circling the same idea without the right words.
"I don’t understand why others didn’t figure this out"
- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
I'm so sorry, in how many?
@mods I am not sure the editorialized title is better! (It is a bit clickbaity, since this is a transformer that does not compete with LLMs at all outside the arc-agi-1 benchmark)
>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
I disagree, but i agree that training with answers is worse.
In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing.
It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well.
It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases.
Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do.
And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other.
Hell, I'm not surprised that the software that the mass of test -studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes
How does it perform on ARC-AGI-3?
There was this a few weeks ago:
"Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163
>> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol
What does that do with 5.6 Luna instead of the expensive models?
What of 'schema' would improve the performance of mdlARC?
mdlARC: https://github.com/mvakde/mdlARC
There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s
> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning
Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.
That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.
The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
[flagged]
Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that?
If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.
Hi! Author here. Surprised to see this on HN now. Happy to answer any questions!
Some context about this:
- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs
- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.
- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)
- Can the perf be improved? Yes but the competition is ongoing so can't talk about it
- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho
- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML