alt Hacker NewsSo, what exactly can a base model do? Is this like the pre-reasoning models that just autocompleted, without any alignment, reasoning, etc optimizations?
Replies
stymaar • today at 1:09 PM
A base model is what you get just from next token prediction training. Think GPT-2/3 but bigger.
Withtout RLHF, instruction-following training, or RLVR to develop reasoning.
A large base model is ~90% of the effort/cost to making a frontier model, and by itself is not useful for much. There aren't many controversial decisions to make when training a base model - you mostly just train a model as big as possible on as much human-generated data as possible.
But it makes it far easier for other parties to do that last 10% to train it to be useful.
However, there are many choices one makes in the last 10% of training which will impact the models behaviour when it comes to AI safety, political opinions, willingness to help make a bomb, etc etc.