Open source is not merely a license choice. It is a reformulation of free software to make it more attractive to businesses. The entire point behind open source is that it is more effective for businesses to develop software collaboratively with the public than it is to do it in private. So yes, open source does imply open community.

If you want to dump code onto the public with a permissive license but not develop that software collaboratively, then sure, you can do that, and the code will be open source code. Opening the code is a good thing and there’s no obligation for you to do anything more. But it isn’t doing what open source was designed to do; it’s ignoring a key part of it.

The people that see open source code and assume that it is being developed collaboratively are not being unreasonable – that’s the purpose of the open source movement. If that’s an inaccurate assumption for your software, then that’s fine – but it’s you that is breaking social norms, not them.