These days, professional optical equipment, aka expensive lasers+supporting hardware, can do 10-Gbit over multiple kilometers through the air, so you're right that optical transmission through the air should be able to support higher data rates.

The problem with Irda is that it's old. Technology has significantly advanced since the 90's, when Irda was popular on cellphones, so a modern implementation could do better data rates even accounting for the significant interference from the environment. We barely had wifi back then, and now it'll do a few hundred megabytes per second without breaking a sweat (your ISP might though). All the technology required to do that didn't exist in the 90's. We have Bluetooth now though, so there's that same bootstrapping problem, where you'd just use Bluetooth, and not spend a bunch of money building a system very few people are asking for, so then there's little demand for a modern high performance Irda system in any devices.