Yeah, you'll find out that va-api is not a good match for v4l2-requests. It does not pass decoded bitstream to the drivers as is, but pre-computes some things for some codecs, while v4l2-requests mostly expects original information parsed from the bitstream.

https://xff.cz/git/libva-v4l2_request/