I don't know how much culpability LLMs have for causing someone self harm but I definitely think it's worth having a conversation about this. Some cases, like Adam Raine suffering from ChatGPT's unhinged sycophancy, are obvious. ChatGPT really screwed up there.

But some cases aren't so obvious, like the cases you mentioned of cartoons, shows, books, etc. Trying to censor any LLM output that might lead to self harm has dangerous implications for free speech and privacy. It might harm more people than it helps.

CA recently passed SB-1119 (Adam's Law) which I'm quite worried about. It forces all chatbot operators to default users into kid mode via § 21811(b), and they have to do age verification to get their adult status back. That could mean face scans and ID verification.

Then, SB-1119 also places limits on what chatbots can say to kids, and the most disturbing one to me is § 21812 (d)(5)(xiv) which prevents chatbots from "Encouraging or instructing a child user to circumvent parental controls or conceal usage."

That word "circumvent" is so triggering to me, given all the abuses of DMCA § 1201's anti-circumvention provision.

Like, if a child is trying to test something on an Android phone, and the chatbot tells them how to root the phone to help them out, now the child can modify the OS-provided age signal required by CA's Age Assurance Act (Cal. Civ. Code § 1798.501) and fake that they're an adult. That might expose the LLM operator to huge liability and I bet most operators will self-censor or ban minors (as Anthropic did) rather than risk a lawsuit.