I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general

I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another

not super pleased with it

the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for