On the flip side, gemini recommended the crisis hotline to the guy.

We can't safeguard things to the point of uselessness. I'm not even sure there is a safeguard you can put in place for a situation like this other than recommending the crisis line (which Gemini did), and then terminating the conversation (which it did not do). But, in critical mental health situations, sometimes just terminating the conversation can also have negative effects.

Maybe LLMs need sort of a surgeon general's warning "Do not use if you have mental health conditions or are suicidal"?

Which is why I love it. It's going to be very disappointing if it gets reigned in just because 0.1% of the population is too unstable to use these new word calculators.