My only experience with VMs on macOS is colima+docker, and it's relatively painful and inefficient (but usable).
Recently got a Mac Mini for local CI purposes (together with Forgejo Actions), took a broad look at the ecosystem and decided to just roll with "build on host" instead. Setting up signing/notarization just looked like an insurmountably task together with isolating it from the host, even with agents. At least the macOS builds are really fast now and the signing/notarization just ~200 lines of Bash...
OrbStack is pretty good. I don't find it inefficient, really.
Try Apple's container CLI. I moved a project of mine from colima+docker to it relatively easily, a couple of weekends ago.
https://github.com/apple/container