alt Hacker NewsContrastive Language Models
Comments
Why is ML so misleading these days? No this model does not score 80%+ on DeepSWE, it merely chooses the best possible idea of Opus 5 at every stage, increasing the performance by 5%.
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
I really hope that "System One" won't stick around as a new buzzword simply meaning "fast".
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%).
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
CLIP[1] for actions! Very cool.
beautiful. would love to see it on some leaderboard
https://benchmarkheaven.com/jev-models
https://huggingface.co/spaces/multimodalart/jev-decision-ind...
Does the latency even matter?
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
Genuine question: Are the use-cases similar to Jev or do you see it applicable for something different? Too dumb to figure it out on my own.
I wonder when work started on this project, and how the public release of Jev played into their timing.
I tried running this on a H100 and got 190ms compared to Jev's 170ms. Maybe I set it up wrong?
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%). [...]
A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...
"On par with Jev" is Mario, T-Rex, WikiRacing: large-K semantic action matching, the regime where cosine over independently encoded vectors is the right inductive bias. The zero-shot suite does not cover the typed-decision load: date arithmetic, negation chains, policy thresholds.
The tech is very cool but for the love of Gaia stop calling it system one, even Kahneman said this (S1/2) is a framework for understanding the brains inner workings. There is no autonomous system to speak of.