To cut through some of all this text - here's a link he calls "the first non-trivial neural network I feel I mostly understand:
https://already-clean.goodfire.com/microscope/medium.html?la...
I think this would be the better link for this
Understanding algorithms beyond a certain complexity is a pipe dream & neural networks are algorithms. The theory says it is not possible & the practice is aligned with the theory. All non-trivial properties of neural networks are undecidable.
I have no idea what I'm looking at here