IBM's PCs were dependable but came at a premium price that was undercut by PC clones. OS/2 might have been superior but it had higher system requirements ($$$) and itself came at a premium price that was undercut by Windows 9x. At the end of the day, it was all about the price.
(Linux, being free as in beer, did the same to Windows Server and the other UNIXes not many years after.)
I wonder if OS/2 was more limited in what hardware it could run on. Hardware vendors made Windows drivers for pretty much everything.
Linux did not substitute for Windows Server to any great degree until quite recently. Commercial Unix, yes.