> Also the whole point of building the program using compilers/libraries in the container is to use such container to run the aforementioned program in that very environment

Says who? That's only because that's the only way it works, not because it's desirable to only work like that.

The only way Apptainers work well is if you put everything you need in one Apptainer, which makes them much less useful than you might imagine.