I agree with what you are saying but this is not how we solved the issue with who can access what.
In our cloud harness we have the notion of shared and private secrets. When something is authenticated with a shared secret it is basically something that can be used in a public setting. If something is private then it can be only accessed via trusted channel. It works for slack, telegram, etc.
In fact, our slack agents are very strong this department and it is guaranteed there is zero leak of private information.
And if someone gets access to a channel that they shouldn't?
The best norms around slack channels is to have most channels be public, but have norms where not everyone is in every channel.