Thanks so much! I hadn't heard of Polanyi's paradox before but it is exactly what I was talking about. The Wikipedia entry even highlights the problem for AI-driven development:

> Polanyi's paradox has been widely considered to identify a major obstacle in the fields of AI and automation, since programming an automated task or system is difficult unless a complete and fully specific description of the procedure is available.