alt Hacker NewsAlthough I’d like this to work, I have not seen evidence that AIs extract good reusable knowledge from sessions without strong guidance.
I do R&D work in comp sci and usually write software and reports or papers in parallel. I have been using Opus in a controlled human-in-the-loop fashion to (significantly) speed up the work. While I’m at times surprised how high-level input steers output the right way, no high level ideas emerge from the AIs output. Many attempts to abstract from the concrete are wrong.
As a consequence, most writing is poor on content and form. It writes about the wrong things and crosses abstraction levels all the time. It cannot keep implementation details and conceptual leaps apart.
I don’t know what this means for the aim of this project to maintain important info for agents. Perhaps there are enough low-hanging fruits. That said, I’m skeptical that the resulting wiki is good documentation for human contributors.
We've found Opus/Gpt are really bad at technical writing. We've found ways around this by having guidelines like https://www.openalmanac.org/ai-patterns-to-avoid.md, which have worked quite well for us.
Understanding what is relevant to be documented vs not is definitely tricky and something we've spent a lot of time on. We found being more prescriptive about what should be documented is helpful, we incorporated guidance from diataxis for it (https://diataxis.fr/)
Also the wiki is primarily for your agents and not for people. Although we're working to improve the prose quality and structure so it is well written for humans too.