alt Hacker NewsReading any analog clock at any time level (edit: and a non-noisy vector rendered image at that) is absolutely table stakes for an allegedly frontier flagship vision model. As much as 1:1 OCR. If the model can't do that, there's something wrong. Doesn't matter if it's memorized some random thing you think is esoteric but is in all the training data and benchmarks.
The whole point of LLM/FMs vs good old fashioned ML is generalization to unknown domains, not just unknown tasks. The hunt for "gotchas" is the hunt for "not in your training data".
Replies
Is this an “alleged frontier flagship vision model”?
This is described as a brand new flash model - still experimental - from a lab that is a side project for an investment firm that has never had a vision model before. That doesn’t scream flagship or frontier to me.
I disagree. It's not even that useful to train LLMs to read an ancient analog clock.
Unless we're talking about AGI, I couldn't care less if an LLM is bad at things they won't be doing anyway.
I'd rather focus training data on more useful tasks.