It was harder to extract a clean signal due to ambient environmental conditions.
You could probably solve those issues with modern tech though. Things have advanced significantly since IR was popular. For example, back then Bluetooth was slow too.