I use kubernetes extensively at work. I don't manage the kubernetes cluster anymore since now we have a team that runs centralized services and you can request a namespace with a quota. But back when my team had a dedicated Azure Kubernetes cluster it was not that bad as people says it is and the biggest hassle was the extremely short lived support for each version.

Then I started to realize most people who complain are rolling their own which is also not bad since there are products like k3s that are very simple to use.

It seems things start to fall apart when they try to stuff it with all kinds of crazy idiotic controllers and the favorite of the month CNI and CSI. I always shake my head when I see people creating sand castles by setting up stuff like Ceph from within the cluster.

If you want to play with it keep things simple and have all the persistent data outside of the cluster. Use good old NFS instead of the latest longceph horngluster version. Keep databases and the container registry out. Treat it like a compute pool not a virtual datacenter. Stop recursing chickens inside eggs.