Is it? I’m pretty sure most mathematicians working on research level math are similarly tucked into specializations. perhaps the only major difference is that in physics the language and jargon differs (and this is purely anecdotally) more across subfields.