alt Hacker NewsThese are thin wrappers over LLMs (specifically, C.AI’s proprietary models). Claiming they are equivalent to tobacco companies fails to understand and in fact minimizes the reality here. If you truly cared about child suicide and weren't just playing up an emotional role, you’d seek to learn how these situations came about.
LLMs are “yes men," and they will on average agree with you with enough arguing. Go chat with one about a fight you had or even make one up and watch as it soothes and supports you. In fact, I put my finished argument into an LLM and opened two different chats and spoke to it in both of our perspectives, and depending on who was prompting it, it claimed the prompter was “right”.
Now, with that in mind, consider the situation: a youth already feels depressed; they talk to the chatbot and, after enough talking, the chatbot is agreeing with them because that is the “probable” mix of tokens. This is platform-agnostic, because it is a fundamental aspect of LLMs. So any value judgment you apply to C.AI specifically must be forwarded to every company, because I have seen, heard, and even made custom roleplay cards with every vendor.
Suicides are unfortunate, but even if the LLMs had a role to play, there’s like what, one confirmed case for C.AI’s entire user base? Even if we assume that was only the “known” cases and assume that there are actually closer to 3000 out of their +30 million user base, then that’s basically the chance of being struck by lightning and is below the standard rate of suicide per the CDC.
There simply is not strong evidence that LLMs are likely to increase suicides, and while they can be a vector for those with underlying issues, so can anything else. People have committed suicide over cartoons, radio shows, books, TikTok trends, and losing game consoles, and certainly not to mention social media.
In short, the advent of LLM chatbots is not strongly correlated to an increased suicide risk, and instead it was an unfortunate situation sensationalized for the eyes of the public by the media. If you wish to push your outrage at anyone, do it at the people who profited off of the suicide and not the company that was picked apart in the process.
Replies
[dead]
I don't know how much culpability LLMs have for causing someone self harm but I definitely think it's worth having a conversation about this. Some cases, like Adam Raine suffering from ChatGPT's unhinged sycophancy, are obvious. ChatGPT really screwed up there.
But some cases aren't so obvious, like the cases you mentioned of cartoons, shows, books, etc. Trying to censor any LLM output that might lead to self harm has dangerous implications for free speech and privacy. It might harm more people than it helps.
CA recently passed SB-1119 (Adam's Law) which I'm quite worried about. It forces all chatbot operators to default users into kid mode via § 21811(b), and they have to do age verification to get their adult status back. That could mean face scans and ID verification.
Then, SB-1119 also places limits on what chatbots can say to kids, and the most disturbing one to me is § 21812 (d)(5)(xiv) which prevents chatbots from "Encouraging or instructing a child user to circumvent parental controls or conceal usage."
That word "circumvent" is so triggering to me, given all the abuses of DMCA § 1201's anti-circumvention provision.
Like, if a child is trying to test something on an Android phone, and the chatbot tells them how to root the phone to help them out, now the child can modify the OS-provided age signal required by CA's Age Assurance Act (Cal. Civ. Code § 1798.501) and fake that they're an adult. That might expose the LLM operator to huge liability and I bet most operators will self-censor or ban minors (as Anthropic did) rather than risk a lawsuit.