If you start with architecture docs that outline the high level goals and what the core abstractions should be for a greenfield project and refine it as you fill in the actual code, you can get pretty good results.

That gives the first few agents the "north star" and then subsequent agents have the actual code, the runtime behavior, and the user experience expectations to work against after that, in addition to the architecture docs, that provide hard boundaries to vibe off of.