In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered.

When I personally want tight code, I have to spend a considerable amount of time adjusting it manually:

- It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating

- I need to consider whether there are solutions based on higher-level assumptions, that AIs typically miss

- I need to check whether there are off-the-shelf solutions - AIs like to reinvent the wheel

IMO, software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.

(1) https://simpsons.fandom.com/wiki/The_Homer

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targeted surgical changes is far more ”productive” in the long term IMO.

I'm probably between $50-$200/day depending on the day; we also have effectively unlimited budget, though a lot of that is because Azure gives startups $150,000 in credits for 2 years, which we've wired up to a LiteLLM gateway & OpenCode. Without that I think our appetite would be more around $400/month/employee.

A lot of my high costs is because I just throw Sol at everything. If I were more selective and brought in Luna or v4 Flash every once in a while, I think I'd be more like ~$400/month. That's why I'm not aligned with the notion that "tokens are subsidized so that's why people are using so much": its not that I'll have to adjust to using less, its just that I'd need to think before I prompt a bit and be more judicious. I could easily see my raw token counts doubling or tripling in the coming months. I don't think that will change as subsidization subsides; though maybe lab revenue will; intelligence per dollar is getting cheaper every week. Its solely a function of adaptation to process, which takes time.

The productivity gains per token are the single most asymmetrical thing I've ever seen in engineering. The engineers on our team are pretty effective with tokens; easily that 2x-4x output as you're seeing, spending $20-$200/day. Some of our security folks have also started contributing more-and-more code, and they're on the other side: they'll spend hundreds a day running in circles, eventually producing these +/-30k loc pull requests that take ages to get merged and are littered with issues. They weren't writing much code before, so arguably they're more productive by some multiplier greater than 1, but I think the drag on the rest of the team, and potential issues with what they produce, has overall created a net-negative situation. Inversely, some other company functions have produced a few one-off websites for things like sales processes, and those have been a huge win. The asymmetry is wild. There's almost a valley of incoming skill where if you know nothing about code, you'll leverage it well; if you know just a little bit, it makes you super dangerous; if you know a lot, you're the biggest winner. Really difficult situation to navigate.

This is very close to my workflow but you forgot one important step:

- Suggest a better approach that makes the AI say, “That’s much simpler. And you’re right. My original plan was over-engineered.”

Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's not very good at testing in an organized way to even notice it made it worse despite repeated prompts to do so I mean if I prompt it to test performance in a handheld structured way (it is very bad at finding out what performance to test and why) before making changes I can usually figure it out but it usually takes insistence on the specifics to really ensure a good solution that will actually fix the problem

You have an unlimited budget, and you only spend $80/day? I’m up to $3k/week, and still expanding.

> Spend most time prioritizing/discussing what to do.

you should probably be doing this discussion work along with Fable 5. It will give good feedback if you're working on the correct things.

> Come up with QA plan and have Codex Computer Use execute on it

QA plan should be part of the above "design + plan", not after it. The implementer needs to be able to fully test before publishing a PR. This is true whether humans or agents are writing the code.

> Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)

unfortunately this is not really scalable with amount of code agents can produce, so you need independent (fresh context) agent reviewers to help. Ideally they only escalate to a human when really stuck.

> I probably spend like $80 a day at least

at a small startup you should be on the $200/month plan(s).

$80 sounds extremely low for what you're describing - are you on API token plans?

I have had some $3,000 token days - even without Fable. I don't see how this is sustainable.

My personal 20x plans get so much usage for so cheap. The consumer subsidies are crazy, but alas I can't use them for work.

Output of 3 or 4 2022 engineers? Its that your self assessment? Output as in number of lines of code?

Only spending $80 a day on Opus 5/Fable 5/GPT 5.6 Sol feels very low. I'll roll through a couple hundred dollars worth of credits a day with those models, the vast majority of which would be on non-coding tasks, and it's still a huge cost savings over me or my team having to do these things manually, if we'd even be able to do them at all.

But that's also why it's now easy to justify the cost of an Nvidia or Intel inference server with Kimi K3 locked and loaded :)

I was just about to say, how could routing possibly be worth it at the risk that the work output is sub par?

The output yes, but do you produce the impact and value of 3 engineers? I have seen this workflow being toyed with too, and I find it to produce massively overengineered stuff that actual people don't really wanna use

My experience is that your description works for a certain time, since you're knowledgeable of the codebase and can guide it. But after too many iterations with not hand-holding the llm, it quickly gets unwieldy.

>Auto-fix bugs and CI until green + thermonuclear review skill x3.

Gotta love this loop, I have it running while I'm asleep all the time.

Do you have tips for generating clean productive output per dollar?

> essentially unlimited AI spend budget

> I probably spend like $80 a day

This doesn’t sound like “unlimited”, I spend more than this out of pocket per day and I have a strict budget.

but I produce the output of 3 or 4 2022 engineers and probably at better quality.

Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.

do the same across 20 terminals (as you should) and now you are up to $1.6k/day. would that give you output or 60-80 engineers? not a chance, right?

no one’s AI spent will be in question working a single terminal with carefully planned out and executed process you do

[dead]