I wish there was some easy way to bet against this happening. I would put a lot of money on the side of this never happening for a multitude of reasons, but I bet I could collect a lot of money from cynics and doomers who think this stuff will happen.

Isn't this how Google operates? I have their AI subscription (about $20 per month). If you want to have a chat history (retain chats after reload) or connect the LLM to Google services (Drive, Emails) you have to activate an option which also allows training. If you don't want to allow training then the subscription is basically useless.

I always pose fundamentalist questions and hypotheticals to the LLM to poison such training data.

Unless your subscription type already comes with a guarantee the data will not be kept or used in training I'd assume the conversations will eventually be used in training regardless how much you paid previously or whether or not you decide to discontinue one day.

I was doing a Udemy course about AI and there was a section where I had to do some processing on randomly scraped tweets and the random tweet that the machine chose to display as an example of something was from a gay porn star and about fisting.

I don't think flat out blackmail will come from LLM companies. It will come from data brokerage companies headquartered overseas.

I'm kind of surprised it hasn't happened already, but I guess there hasn't been enough unscrupulous LLM companies selling those "anonymous" chat logs yet.

> they will be public as paid training data.

Your data is already training data. If they promise to delete everything from their models or those elsewhere that they made the data available to, even if you pay, I'd call them liars.

> (...) your anonymized chats, they will be public as paid training data.

If they are PII then under GDPR they are obligated to delete the data.

If not then they will be liable to pay fines up to $20 million or 4% of their total global turnover.