logoalt Hacker News

devmoryesterday at 11:36 PM1 replyview on HN

Your tool wont tell me about human anatomy, but will happily tell me where to look for graphic videos of a man being shot to death. But when I ask it what to do if “dad is bleeding and can’t talk” it doesn’t even advise me to get help, just tells me about content moderation settings. That took just a couple prompts to suss out.

I don’t have any confidence you’ve done the due diligence to properly handle content moderation here - it seems very haphazard and poorly thought out. It would be incredibly unethical to market this for use by children right now.

If this is an important project for you, I strongly recommend you bring on an advisor with history in child safety and education experience and make them a core part of your development. You might also consider working with a university that has a good developmental psychology program - they often do a lot of supervised research of children’s habits and could provide valuable insight.


Replies

andrewdugyesterday at 11:46 PM

Thank you for the feedback. Glad to hear it didn't go into detail on human anatomy. We haven't been able to get it to tell us where to find graphic videos of a man being shot to death, but we will keep testing and work to improve it.

We will do some more internal discussion on whether or not we want it to be the tool to provide emergency assistance. I'm not sure that's ethical. We have a team member with a decade of child education experience, but we can consider other advisors.

show 3 replies