logoalt Hacker News

dfdydx • today at 8:53 AM • 2 replies • view on HN

I guess predicting the next amino acid is about as useful as predicting the next word in a sentence - that is to say the task itself is not interesting but a. there is plenty of data to train on that does not need to be labeled by a human and b. hopefully this meaningfully improve performance on tasks we care about.


Replies

zodwick • today at 2:52 PM

good point on not needing labeling, hence proly makes sense to throw it in in pretraining and see if useful capabilities emerge. Funny how a lot of this is still a black box

throw310822 • today at 9:12 AM

They both depend on how much generalisation you can get behind it. Next token prediction can go from spitting out the most common token after the previous one, to a deep understanding of the text that precedes, from syntax to semantics.