alt Hacker News> I think the more productive way for frontier models to interact with a unity project is to directly manipulate the raw scene files.
I disagree. Even moreso than webdev, gamedev benefits from a fast and reliable loop, especially one that can capture realtime state and screenshots. It's the difference between having a new well tested mechanic in your game next morning and giving up after a week of debugging.
Agents will work around slow iteration loops perfectly fine given enough time, but when you're shipping 20 core things in your game each day, the agent's test harness becomes the bottleneck.
IMO it's worth it to spend a day or two to make sure your test loop is tight, it will pay dividends. Not saying Unity is great at playing ball here though. Hopefully their new effort helps.
Letting the agent take screenshots of the scene is not effective. A human needs to be in that loop.
You can get primitive layouts working if you feed it orthographic projections from the 6 faces, but it falls apart once you need to start looking at things in perspective. It's also absolutely ass at interpreting scene illumination.
We need models trained on 3d scene transforms (aka world models). Passing raster images into the model is a really inefficient method of sending this logical information.