alt Hacker NewsCost per task, time per task, quality (good luck measuring objectively between models) are the metric that matter. I don't care how much a token costs, I'm not buying tokens, I'm buying solutions.
Replies
agoodusername63 • yesterday at 11:31 PM
the point is that "task" is a dubiously defined unit that the frontier labs adjust to whatever makes them look good.
mapontosevenths • yesterday at 9:28 PM
Are OpenAI's models delivering 50% (or more) completed tasks than they were last month? Are they delivering 50% more than Anthropic per token?
Nope to both. Not even close.
The only way it works is if you define "task" in a way that OpenAI is good at it and Anthropic isn't.
➕ show 1 reply
Exactly! I called that the Operations Equation and wrote a tiny bit more here about it: https://zero2data.substack.com/p/the-operations-equation
Once you are inserting models into non-creative (and software development is creative work) it is imperative to start creating your own small evals. Not just to check how different models perform in that equation, but to catch model drift over time.