I've got worker nodes on m920qs and proxmox VMs too for workers, but like the clean segregation of a HA control node group.
I doubt it matter much either way tbh. K8S is inherently resilient.
I did have the rasps on hand though...not sure I'd go and specifically buy them just to achieve said segregation.