alt Hacker NewsNow I’m wondering what is the eigenspace of an LLM? If I take a set of LLM’s with the same number of parameters, then what are the eigenvectors? Do they have different personalities?
Now I’m wondering what is the eigenspace of an LLM? If I take a set of LLM’s with the same number of parameters, then what are the eigenvectors? Do they have different personalities?
Neural networks are non-linear, so I think you wouldn’t be able to compute typical eigenvalues. You could compute the eigenvalues and/or singular of the individual weight matrices (I’m sure this has been studied). SVDs are very conventional for making low-rank approximations, so it must have been studied.
The concept of nonlinear eigenvalues exists, but it is a bit more exotic.