Yes, but its gonna be weird. I've done very tangentially related work with game controllers, and my suspicion is you're gonna have to try a few (dozen?) before you find one that even sends reasonable commands. I can imagine a FireTV stick sending game controller inputs. I have no idea, but that's the type of thing that might go wrong.

A MX3 air mouse might be the exact thing you want though.