When you open a short position you borrow shares from your broker's other customers. Your broker then sells those shares on the exchange. When you close your short position, your broker buys shares in that same stock from the exchange at the current price. So the people losing money are the people who bought when the market was high and sold when it was low, as always.

There are two big issues with shorting a stock. One, your downside is infinite, whereas your upside is only the size of your position. If you short a medical stock worth ten cents and it zooms up to $1000 because the company discovers a cure for cancer, that's going to cost you $999.90 for every share you shorted at ten cents. If the company goes bankrupt instead, you make... ten cents for every share. If you get unlucky a single short position will wipe out all the money you made or will make shorting stocks for the next three generations.

The second problem is you don't completely control your position. If you buy a stock to hold, it's yours until you decide to sell. But when you short a stock and enough the people at your brokerage holding shares in a company you shorted decide to sell, your broker will summarily close your position at the current market price because there aren't enough remaining shares for you to keep borrowing. That can be very frustrating if the stock is at a temporary peak, especially if it proceeds to go down to a price for which you would have closed at a profit.

EDIT: I suppose I should add a third problem to the list. If the cost of your short goes beyond a certain percentage of your account your broker will close your position to protect himself and his other customers. That usually happens if the stock is going up quickly. When your broker closes your position, he, along with all the other brokers closing short positions, needs to buy stock, which creates a positive feedback loop. That's called a "short squeeze". You can end up with prices shooting up to ridiculous levels because people have no choice but to buy.