Yes LLMs fail and fail often. I use a council of 3 for my health related conversations. They have recommended stuff which proved useless, I even wasted a little money in supplements which later iterations of the same LLM explained are not useful at all. I failed to explain to the LLM that some of what I was taking (baby aspirins) was not a doctor's prescription but my own initiatives, so it didn't dissuade me from continuing them until it came up much later - something my own doctor also agreed should be dropped - she had also assumed it was something I had been prescribed in the past!

Regarding Gell-Mann amnesia, I literally mention it in a separate post in this same thread :)

And I'm sorry if your experience is that most medical professionals would react like that. This isn't absolutely the case for me. I show up with things printed out, don't come across as a lunatic who googled a symptom and just wants to be agreed with, and I explain that my LLM research is so I ask fewer stupid questions in the limited time the doc has for me.