Sure, in X11 https://www.x.org/releases/X11R7.7/doc/xextproto/shape.html

And for Wayland I found this

https://dev.tizen.narkive.com/AkKHZRHh/wayland-non-rectangul...