It's an undertaking, but there were multiple X servers for small platforms back in the day - the difficulty is overstated. It takes a few thousand lines of code to get a basic one together.

Maintaining Xorg might be unreasonably hard.

But I agree with you LLMs are amazing at this - I had Claude throw a half baked one together in less than 24h.