It's more like the corporation must because it's fully capable of doing it, and it's profitable, and there are shareholders to answer to. They watch you either way, but right now it's only for their own benefit. The corporation doesn't care how many people are harmed by what their chat bot tells them. It would cost them money to try to prevent those harms, so they haven't really bothered to beyond some half-assed token efforts intended to keep more costly regulation at bay.

Big brother watches you because we know he does and the incentives are set up to make watching you profitable?

If big brother wasn’t watching you while he subsidizes your use of his tools, then he is leaving money on the table. Which means he will get bought out and replaced by a big brother who makes the quarterly numbers go up.