As far as I understand it, the main problem with keeping Xorg going isn't anything inherent with the code itself (although that's definitely a factor), it's the sheer amount of functionality Xorg clients can access, including viewing and manipulating global state and other clients. When you combine this with X11's ability to remotely view clients from other devices, and the fact that most of the main Xorg maintainers are paid employees of various commercial Unix and Linux vendors, it makes change extremely difficult. You can't try to do things to modernize Xorg, like making the server multithreaded, changing how the graphics work to better fit modern display hardware, or adding a permissions mechanism, if you then have to deal with a bunch of urgent support tickets from large customers complaining that your changes have broken some 30 year old software they rely on that they have running in a VM somewhere.