alt Hacker NewsIntroducing System One Models and Jev
Comments
You know you're too old when you see the company name and think! Oh I wonder what Martin Odeskey , Jonas Bonér and co are up to. Wait, didn't they become lightbend... Altho this comment takes away from what these guys are doing which legitimately sounds interesting.
Amazing work by the team! Looks like they've traded accuracy for speed and this is most likely going to be the case with the next class of models.
This is a valid tradeoff for one-off responses but if we're dealing with a distributed system (eg: Kafka), then only the high-confidence responses (>0.8) should move forward as input to the next service. If a low confidence output is propagated, then it can break the entire chain.
So how do we evaluate how good this is?
Couldn't you hook it up to a multiple choice exam?
It's still LLM like, how smart is it? I'm wary of something that the company states they don't want to benchmark it across public benchmarks.
From the person in the video regarding issues with benchmarks in general, and for LLMs. Also their approach. Good article.
The fact this isn't open-source is troublesome.
Such large advances shouldn't be locked up away from local hardware.
Great to see something new..
However I don't understand how are they claiming zero hallucination, how does giving confidence score fix hallucination? or am I missing something here?
Interesting. Perhaps I can see this being quickly adopted in LLMs-as-a-judge, where you normally need (a) a structured answer, say, with lots of different fields (metrics) and (b) you want the judge to be fast, not being a bottleneck.
I'd love to know if Jev is still fundamentally LLM-shaped in architecture. Like is it using a single forward pass with a learned readout over the predefined options (i.e. a discriminative head on a transformer, no decoding), or something else? I did similar things for zero-shot criterion-based classification using a 4B Qwen model but could not reach the level of intelligence they've got here. Tho speed/cheapness was similar.
I was going to ask if the inspiration for System One name came from Kahneman's and Tversky's research, then I read the FAQ.
I listened to "Thinking fast and slow" recently and I was surprised how closely in behavior a LLM approaches the "System One" as defined there.
I approve of the clever branding!
Congrats on the release!
Finetuning a language model for decision classification (with probabilities) is already well-understood. What specifically changes in the training objective with RLCD? Are its benefits isolated from Jev’s new architecture/parallelism?
One application that sounds pretty interesting would be the creation of wikidata pages for anything. Plug a topic/word/concept/historical event in, take a bunch of wikidata properties, rephrase them as questions with the choices being the existing property values. Then feed it to LLMs or something. Does that make them more reliable? Probably not.
Looks promising. I'm building an AI video editor and multi tool calls take >30s using Gemini. This would be a a game changer if Jev can take that down to single digits at p95.
Very cool, I immediately jumped on the waitlist and shared this with my co-founders. Any plans for offering this through a European provider at some point after launching in the US? We work in EdTech, so non-EU-sovereign solutions are a harder sell to our customers.
This will be insane for tool usage, and probably where the major economics for day-to-day usage will be.
The goal is going to be to use llms to distill operations down to some dsl, and pass it into something like Jev.
If we could come up with a system to classify the probabilities across a large number of candidate words (or components thereof) then this could actually be good at producing text, one element at a time. We could call these elements 'tokens' and picking the right one could be called something like 'decoding'. Crazy idea but hear me out...
On a more serious note, it will be fascinating to see how this different spin on modelling inference will create new paradigms or slot into existing ones.
Hm, would be good to understand the architecture better. Is this answering just from a world model informed prior? How informed is it by the information in the prompt? I can't see this maintaining calibration across all domains and all types of structured output.
Is there anything published on how it maintains calibration? Or when you say "outputs calibrated probabilities" you mean "as calibrated as frontier LLM models, just cheaper" - which is a different claim; as LLM's aren't particularly well calibrated
Super cool! Instantly joined the waitlist.
It might be boring, but I can see exactly how I could use this right now to improve my agentic rag.[0] In two months I am supposed to deal with a giant corpus, while still maintaining responsive chat UX. I have been working my butt off to make our first big client happy. This could really help solve the chunk ranking problem.
[0] assuming the policies are compatible with sensitive production workloads, some time in the near future.
I can see the value in this but looks like there's going to be trouble in communicating the difference between this and a regular LLM, and also proving the potential cost savings in using this to replace existing systems that are using LLMs with frameworks like langgraph, as this can't be a drop in replacement and would require a significant amount of re-architecting/reengineering of systems to get the type system to work
Interesting concept, I can't see a reason to use a generalist classifier over an api rather then just training my own? If it was open weights I would probably mess around with it.
Hasn't there been a lot talk about Astra's opaque reasoning capabilities (being able to think through complex questions without using a chain of thought)?
Given that, can't you just replicate Jev by telling Astra "here is the question, you must make a multiple choice decision / output a score between 1-10, please answer directly in a single word, no reasoning allowed"?
(Edit: Ok, Jev is much cheaper in input tokens so these two aren't directly comparable at all)
While I understand that accelerating development isn't necessarily the target for this, and it's not at all intended to generate code the way many of us are...
I think this could be pretty decent in CI? There's a lot of "flakes" I've mediated that this could have handled much more efficiently. Maybe observability as well, triggering elevated logging and other initial measures?
I'm trying to understand what difference does this make over LLMs.
LLMs are universal simulators, their latents model the world. So I bet if you compare their logprobs with probabilities output by this model, it will be highly correlated.
Someone should do this quick experiment. I bet there won't be enough of a meaningful difference.
do you all see the use cases being similar to what you might use Fastino's Gliner models for? i see similar differentiation from general purpose LLMs in the sense that they can take natural-language input and return outputs adherent to a user-defined schema.
https://fastino.ai/blog/gliner2-5-span-free-information-extr...
im thinking about how well Jev could be used to replace a current LLM-as-Judge evaluation workflows, specifically on chat transcript data (think ~1,500 tokens) i wonder if the reasoning usually required pushes it a bit out of scope. didnt see anything published about constraints on the state size, so would be curious to hear about that.
This is basically a zero-shot classifier that can accept raw text (or structured text) as an input, and is able to classify that text as accurately (they claim) as a frontier-level LLM. I have workflows this would be useful for, looking forward to it showing up on OpenRouter.
This is amazing. I really could use this.
I like the idea of System one models but all LLMs so far work as system 1 thinking because humans generate speech subconsciously with system 1.
System 2 thinking requires consciousness which AI does not have, so even reasoning models are still system 1 thinking as system 1 in humans has reasoning with heuristics.
Its limited but most people navigate the world with it completely, so it's enough for AI.
As a zero-shot classifier, I expect that effectiveness is dependent on the data trained upon.
Jev input … > Unstructured data (e.g. text) with an emphasis on structured program state.
What pre-training data/model is Jev based on? Surely result effectiveness is dependent (outside of one’s own input as “state”) on that?
A question I have, with the type { output: string }, would the model not become a LLM? And if it does, shouldn’t it cost the same as a LLM for output?
Super cool. Does it, or will it, work with image, audio, or video input?
What are peoples' thoughts on whether a local version of Jev is possible? Having to call an API for something that's main benefit is speed is orthagonal to their ethos.
Always exciting to see people working on novel models, rather than the Nth version of the same slightly tweaked LLM.
I'm very curious how much ressources are needed to run such a model. This could be a complete game changer for local applications.
Overall this seems like a classifier that gives weighted scores per custom labels. It's certainly useful, but whether it brings higher quality than an LLM in structured output mode has to be seen in objective benchmarks.
This makes me think of Expressions of Change [1], a project that aimed to make updates to a program a first-class primitive in a programming language. A model like this can't output code directly, but perhaps it would be well suited to select from the small set of discrete operations on code envisioned by the EoC author?
[1]: www.expressionsofchange.org
Someone else noticed the base64 encoded block on the launch site[0]? Its the fast inverse square root algorithm q_rsqrt[1]. I guess its meant as a joke to put this algorithm that makes use of type punning on a site called typesafe.ai? Or maybe because of its efficiency?
So is it a structured data-based language model? Or is there a model and a harness? Hopefully they’ll open up and explain more.
Signed up for the beta! :) would love to put this through some real-world shootouts against traditional LLMs to see where this type of model really excels.
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
Congrats on the launch! What's different between Jev and Microsoft's Guidance package? https://github.com/guidance-ai/guidance Is it a diffusion generator under the hood?
This sort of stuff almost sends shivers down my spine, it's like i'm looking 5 years into the future.
A few questions:
1. Do you provide any kind of largest common subtree caching for cheaper input?
2. Have you tried auto-generating Lisp programs structurally?
3. Have you tried augmenting a Lisp language with a `choice` function that makes choices given a prompt, the environment, and the continuation stack?
Seems like LLM can do everything Jev can do (just structured outputs?) but Jev is highly optimized and purpose built for it and thus way faster and cheaper. Is that a fair description?
Funny how the authors are asserting that "doing the right task > data > compute > algorithms" while simultaneously releasing AI model for calibrated decision making, which if they work, would mean that "compute > doing the right task"
Wild that it doesn't generate text. I wonder how its technology compares to Tesla's FSD stack.
I am positive I know exactly how this works, I made something similar a few months back. But the problem is without generation you are extremely limited in the use cases. And while the model can't hallucinate, it can still be wrong. It just can't make up data.
> We deliberately chose not to publish performance against public benchmarks. In fact, we plan to only have one-off evals when we make product updates.
lol, I bet they would publish them if their score on those benchmarks were good.
How does Jev compare with encoder language models like BERT/RoBERTa/DistilBERT, which could also be used for text classification?
I was thinking about something similar (maybe) - generally speaking, embeddings for LLMs tend to learn real world concepts - things like 'fruit' or 'France' or 'city' as directions in embeddings.
But in things like programming, most concepts are abstract - 'if hungry eat an apple' in programming terms would look like
'if hunger > 50 {apples--; hunger-=30;}'
and compilers work with 'concept erasure' - to them, tokens (which are like llm tokens) look like
'if var1 > 50 {var2--;var1-=30}'.
They don't care about how these things map to real concepts. So all the embedding directions used to encode real-world concepts are just noise to LLMs when programming. This greatly reduces dimensionality and training costs. So does a token representation tuned for programming constructs, rather than natural language would probably have a more efficient encoding.