I messed with this a bit and found fbdev forces even gpu accelerated video decode or rendered apps (e.g. even terminal text these days) to go through the CPU/RAM copy path whereas a lightweight graphical setup (like a minimal sway with a full screen terminal window open) is zero copy. Minor thing, but I'll gladly trade a little RAM for some battery life + simplicity if I ever do need to open something actually graphical on the machine.

I'm curious, what's the motivation for not having a graphical environment at all?

It's not an answer to your question per se but I find various keyboard driven tiling WMs to be sufficiently minimal while still allowing GPU accelerated stuff to "just work".

Don't use mplayer for this. Use mpv --vo=gpu --gpu-context=drm

This gives you hardware acceleration without a desktop environment.