I think it's reasonable that humans need to provide context for this, and situations like this, forever. The GP has a specific problem. The context of the problem exists in the real world, not something that the agent can search for. This isn't a "humans are still important" take. This is custom software for meeting the specific needs and preferences of an individual.
I fully agree with them that the easiest solution is to ask an agent for exactly what you want, and iterate from there. Evaluating and installing and learning off-the-shelf software is often more of a pain than doing it from scratch.
What prevents a solution where you physically hook up the needed components and then type "/goal make this keyboard extension work"
And the same library is built without the "seed" or baseline?