alt Hacker NewsWow. Are you saying you trained a smaller model from scratch (from what training data?), then used a larger model (which model?) to tweak its weights to fix this problem? Or did you post-train/fine-tune an existing model? I'm very curious what models, hardware, and training data you used!
> Are you saying you trained a smaller model from scratch
Not yet - but I really want to. Current experiments are on Qwen3.5-2B, and I've done a bunch on ModernBERT-large.
My main hypothesis is: I'm fairly confident I can come up with a university curriculum for a high school student to learn certain skills so that even without gaining technical knowledge, they can call bullshit on commit messages written by Claude and Codex. And I'm basically converting that curriculum into training.
Training a model from scratch is likely the next thing I'll do once I hit a significant bottle neck. There are still quite a few things to try (ex: Franken-merging, converting the dense model into a MoE, etc).
> very curious what models, hardware
Mostly using run pod and similar, and honestly, grabbing any decently sized GPU that I can get my hands on. 5090 - B200s.
> training data
Public data and synthetic data. This has been the hardest part. Also, why I'm excited about not having to "train" the behavior into the models.