Same, years ago, I set up a Matrix server, because it was advertised as the-new-XMPP (and I had done XMPP as a user, a long time before, and thought it had been quirky enough to warrant a successor protocol).

What I found with Matrix was the same terrible experience you describe, so I gave old XMPP a new look, and it's been great and continuously improving since. I sleep much better at night having my whole family using XMPP over a self-hosted ejabberd than I can using Matrix to talk with them (and synapse... Forget using synapse federated).