logoalt Hacker News

maxgee • yesterday at 6:45 PM • 20 replies • view on HN

Our account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.

We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.

OpenAI does not care about fighting child sexual abuse, this is completely performative.

EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com


Replies

uptrails • yesterday at 7:13 PM

I've done some work in this space and it's mostly that Big Tech has regulators and parent advocacy groups already very angry with them and breathing down their necks, so they would rather lose customers than take a nuanced approach. The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.

Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.

I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".

➕ show 2 replies
icantevenhold • yesterday at 6:52 PM

I would posit almost no one really cares about fighting child sexual abuse; most of them just use this as pretence for more surveillance and data collection.

➕ show 7 replies
mattmaroon • yesterday at 6:56 PM

I’m sorry that happened to you, but I would read this more as they are an organization not prepared for this. Why would they be? One minute you’re trying to make computers pass the turning test, the next minute child porn is getting involved.

The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.

➕ show 2 replies
Jimmc414 • yesterday at 9:04 PM

I had the same experience with OAI. We were all-in with OpenAI as our primary provider. We used it exclusively for python and go programming requests. Received a ban for "inciting violence". Appeal denied, no refund, no human response or explanation, 3 years of data lost (privacy requests for data denied). We're on team Anthropic now. Still stings when I think about it.

➕ show 2 replies
kkukshtel • yesterday at 9:59 PM

I run a moodboard website (sort of like pinterest, but no social features) and I have been worrying about this as well. The site is picking up pretty decent traffic and has hit nearly 1TB in uploaded images, but I haven't yet started scanning incoming images for CSAM content.

Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).

throwaway81523 • yesterday at 9:48 PM

If OpenAI actually flagged something as CSAM, they were supposed to report it to NCMEC. I wonder whether they will confirm having done so, even if they won't say which image it was.

I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.

nonethewiser • yesterday at 6:54 PM

> OpenAI does not care about fighting child sexual abuse, this is completely performative.

That doesn’t really follow from your experience.

OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.

➕ show 4 replies
trollbridge • yesterday at 10:46 PM

Thanks for running ytmnd. When I was a child, it brought me a great deal of (juvenile) amusement.

Duck roll…

delichon • yesterday at 7:35 PM

Classifying unreviewed user-uploaded content with a service that does not accept CSAM is a footgun, even if they misclassified something this time.

➕ show 4 replies
chinathrow • yesterday at 8:40 PM

If you throw huge amount of work loads at an LLM and expect 100% correct work in order not to loose your account, you might want to reconsider your expectation, no?

pluc • yesterday at 10:25 PM

Performative would be giving the people (everyone) the means to expand the crime they seek to punish

randbyte • yesterday at 9:12 PM

Performative and just try to shed liability.

NooneAtAll5 • yesterday at 10:38 PM

why even have openai account then?

stay away, use deepseek and stuff

cyanydeez • yesterday at 8:52 PM

it's not performative if it's done to reduce or eliminate liability. Performative is about signally for the purpose of signalling. Businesses dont signal for it's own sake. They do it to either increase business or decrease liability.

surgical_fire • yesterday at 7:35 PM

Oh my, ytmnd!

I used to view shit there a lot sometime in the mid-2000s.

teiferer • yesterday at 6:54 PM

Seems like they want to play in the big boys club (Google, Facebook, ... ) where this kind of treatment is standard.

ajjahs • yesterday at 6:52 PM

[dead]

gabucino • yesterday at 8:26 PM

[dead]

jstummbillig • yesterday at 8:40 PM

Imagine you are OpenAI, with hundreds of thousands of business customers and trillions of actions taken through your system every day, in violation of your policies and often local laws.

How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?

➕ show 2 replies
antiloper • yesterday at 6:55 PM

I mean, you're holding it wrong. You uploaded CSAM to OpenAI which is presumably against their TOS.

➕ show 3 replies