I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
My main takeaway is that HN had no idea the field of interpretability existed until now
I don't know why because Anthropic has had a lot of blog posts about internal work on it