This is sort of a solved problem with virtio native context. You can just run mesa driver in the virtual machine and proxy it's ioctl calls to host kernel gpu driver.

I'm pretty sure all the major GPU drivers in mesa now supports this feature.

Wonder why QubesOS hasn't adopted this yet.