Managing AI Coding Costs at Scale

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like:

- Spend most time prioritizing/discussing what to do.

- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)

- Use Opus 5 or Sol Med to execute

- Auto-fix bugs and CI until green + thermonuclear review skill x3.

- Manual interrogation of change/nits

- Come up with QA plan and have Codex Computer Use execute on it

- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)

I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year".

On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.

Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.

I suspect that when it comes to hard complex software products, you’re better off ignoring agents and doing “trad coding”. What you lose in short term speed you gain in manageable complex codebases.

If you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term.

Now of course, there are products that just involve lots of code but are not actually complex. This is generally the project with like hundreds or thousands of features but most of the features are separate and don’t actually interact in complex ways. Think a task management app with hundreds of features like calendar, email integration etc. there I think agents gives you more bang for the buck. Just my thought, using agents at work.

Careful. If you admit to using models that weren't trained by OpenAI or Anthropic then you might hauled in front of Congress: https://www.scmp.com/news/china/diplomacy/article/3362616/us...

I find this funny and interesting at some levels

1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes

2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model

Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.

This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs.

Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity

I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)

> nearly every company deploying AI tools at scale has hit the same wall: exponentially growing costs. That curve is unsustainable - left unchecked it will eventually overtake revenue.

But the question remains, AI hasn't shown any explosion in growth/revenue for most companies. The effects seem muted.

What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant.

One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)

Surprisingly pragmatic and info packed article..

Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools.

I think building companies is going to look more generic in the future because intelligence is an API now.

It’s funny how different everyone’s experience is with this stuff. To me the diminishing returns are more around not going crazy with prototyping or running with xmax thinking all the time. I haven’t found it hard to stay under the usage limit of one $200/mo Claude and one $200/mo Codex subscription.

If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.

As a solo dev, this gives me hope.

I feel like I have an advantage over big companies, if I can use the best models on a subscription and not worry about costs much, when they can't do the same as outlined in the article.

I authored this - happy to answer any questions.

> Rapidly adopting newer, more efficient models delivers the largest cost wins of any technique.

I think the more interesting lever is the fourth they mention: token efficiency.

> By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include.

I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.

First time hearing of Omnigent. Anyone have experience using it?

These seem like the obvious tweaks akin to "using a cheaper hosting platform". I think the real savings come from careful context control for programmatic agents, careful tool awareness and usage to reduce thrashing, distilling workflows into deterministic processes and, moat importantly, adding friction and boundaries for non-technical users who tend to burn tokens making insane asks like "analyze all documents and give me a summary".

Reads like an add to Omnigent or whatever harness (wait it’s meta harness?.

How can Smart Router achieve higher task completion rate compared to any of the base models if all it does is dynamically switch base models based on cost??

not sure about the use of exponential and efficiency frontier here, these have formal sides to them but seem to be used rather inflationary and colloquially.

“use lower cost models”

“use price controls”

Truly revolutionary stuff.

Appreciate the detail in this and the previous post on creating internal benchmarks!

Have you all attempted finetuning smaller OSS models on your repos for coding?

I think there is a lot of dev cope in this thread.

My workflow is very simple:

1. develop requirements for code change

2. take manual notes for implementation, maybe use LLM for some discovery/investigation

3. present notes to frontier LLM

4. develop implementation plan (bulk of work)

5. let LLM rip

6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions

7. get automated LLM review

8. get human review

this reliably produces the work of 2-3 pre-AI senior engineers with a lower bug rate, equivalent performance, robust edge-case consideration, etc.

Does the LLM produce over-engineered solutions? All the time. I stop it from doing that, or manually fix it myself.

Does the LLM always adhere to the best system design? No, not at all. I often have to guide its design into a better, north-star aligned one.

I don't just sit in front of my terminal and say, "Ok Claude, build the app." It is a very iterative process, and not without its potential pitfalls.

But it is very, very productive.

how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)

good engineer + llm = good engineer.

bad engineer + llm = bad engineer.

is this opensource or have to buy from Databricks?

So did we. I just asked my team to get personal accounts that I reimburse them for. It’s just a golden age loop though, the gravy train can’t go on forever unless we start building out thousands of data centers and associated renewable energy.

First the mofos force you to use AI then they become stingy about it.

An AI-edited post by the way.

Probably coulda got every dev a local model for how much they spent; what a brialliant set of economists

[flagged]

Really? Because removing it from my company has saved us over 2 million a year and we were able to speed up processing. The chargeback model for databricks is predatory at best.

Yawn. Databricks and their half baked overly expensive platform.

Too bad their AI query generation is next to useless.

Quit cold turkey and you can drive down AI coding spend 100%.