You can be productive and guide AI to operate in a more compliant way: by writing down basically everything you know, every heuristic you have, documenting the correct patterns, documenting the framework level approach, explicitly spelling out what's good code, writing down detailed specs for an agent to work on, writing the test plan, etc.
A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output.
A concrete example: I wrote a small library[0] (MIT, OSS) that improves how agents write unit and integration tests. This isn't product code, but this makes the agent better at writing the tests it uses to verify its work. (Tests become denser, more semantic, less context, type-checked setup, etc.). You can't expect an agent to do this by itself; the human needs to produce the scaffolding.
agree- this is how i name variables, this is how we handle third party APIs, this is how we handle caching and here are some heuristics for when we want to cache and when we want to avoid it, etc. etc. all coded up in a nice documentation tree with links in the agents.md that say "when integrating with third party APIs, read docs/integrating-with-third-party-apis.md".
i have written alot of this type of documentation as an architect for human teams, and i find it works for agents as well.
Do you have eval testing? Benchmarks? Deterministic model checkpoints? Deterministic harness versioning?
Building "orchestration" without being able to deterministically measure improvements is not engineering, it's cargo cult.
> A lot of devs still think their job is to produce code; it's actually to produce all of the guidance and structural scaffolding for the agents ("the harness") to guide its behavior and output.
This basic reasoning is wrong and leads to “orchestration porn”. Thinking that you can laid down rules to automate software production like it’s a factory pumping out products.
What you forget is that software are grown iteratively, not guided down some assembly line. Every patch is a revision of what the software should do, not the next step of a workflow.
So stuff like patterns, architecture, tests,… are helpers for communicating the purpose of the software. They’re not really relevant to the software itself as the computer would be happy with just a single file.
You have a software that’s needed to fulfill some needs, you have a team of people where each brings expertise to build it. Workflows are built so that everyone how to contribute, what they can depends on, and what others need. When there’s a failure, we add the missing information. When there’s need to do some changes, we inform everyone impacted.
Everything is information and communication related. Information about the system and communication between everyone that interacts with the latter.
So devs do not think their job is to produce code. It is to take requirements written in ambiguous natural language, sometimes incomplete and contradictory, and to design a solution that is formally written while taking care that all statements are coherent.
Making things coherent (aka correct) is what takes the majority of our time. The other is gathering requirements. The actual typing of code is pretty much instinctive, just like you don’t check the orthography of words like “concrete” or “guide”, and rely on the spellchecker for more difficult one.
The “orchestration porn” part is that there’s no mention anywhere on how you think correctness and be improved or even maintained as part of your tool.
[flagged]
> You can be productive and guide AI to operate in a more compliant way: by writing down basically everything you know, every heuristic you have [...]
If its ends up being really that straightforward, it sounds less like engineering a product and uncomfortably close to building your own to corporate-owned replacement so that you can be fired.