Actually most watch collectors do not wear their most expensive watches, they have daily or travel watches to wear that carry less risk.

Also, watches are status symbols to a really narrow niche. Vast majority of people cannot name a non-Rolex expensive watch and would assume Seiko (and maybe Tissot) are the best watches after Rolex, followed by Swatch (and maybe Timex).