logoalt Hacker News

ritz_labringue • yesterday at 6:28 PM • 3 replies • view on HN

We've moved past per token cost a long time ago. Smaller models use more tokens. What's interesting is the cost per task, on that front deepseek isn't particularly good.

https://artificialanalysis.ai/#intelligence-efficiency-tabs


Replies

bscphil • today at 12:14 AM

The AA benchmarks seem to be mostly "complete this small intelligence-proving task from scratch". That consistently puts models with high intelligence to output tokens on the frontier, especially the OpenAI models, on the frontier. But a lot of software engineering work is many-turn work on established code bases. There's a lot of reading and a relatively large context unless it's carefully controlled. Writing only a few output tokens doesn't really cut costs because most of the spend is read tokens that get recycled through dozens of turns. That's basically the ideal case for DeepSeek because of how cheap cached tokens are.

joshheitzman • yesterday at 6:43 PM

It highly depends on the task. So far, on the tasks I personally benchmark, DS v4.1 Flash @ deepinfra.com is always cheaper because their cached input price for it is so very low and their cache hit rate is so very high.

gregwebs • yesterday at 9:54 PM

That analysis is unfortunately running DeepSeek on Max. DeepSeek recommends against that since it guarantees higher cost and rarely improves the outcome- it should be ran on "high".

➕ show 2 replies