Side loading is an interesting hobby horse for hackers. It causes material harm to a lot of people. But hackers want to keep it anyway for themselves for ideological and aesthetic reasons.
Ideological is carrying a lot of weight there. Perhaps you can be more specific about the ideological arguments you are hearing that are not worth it?
Citation please; and remember your answer is incomplete without a comparison to the safety of NON sideloaded apps.
Okay, then every book, every email, every text message, every comment, and every letter should be signed by a third party that's verified your ID. After all, there's speech which can cause material harm and free speech is just an ideological thing. It'd be dangerous if we allowed unsigned messages to be sent between people.
Who says that Google is the one to decide what open source software I can install on my mobile Android computing device?