Not who you responded to, but I’ll give it a try.

Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):

0.728201017

-1.9293839303

2.93020202929

5.82290101728

-0.2844920917

…

And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).

Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:

0.428201017

-1.6293839303

2.63020202929

5.52290101728

-0.5844920917

…

(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)

Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.

Please do correct me if this simplified too much or I had a mistake in my understanding.

Only a tiny, tiny fraction of the parameters are encoding information that's specific to a particular programming language. Even if you could remove those without degrading performance, it would have a negligible effect on the model size.

For a set parameter size, if you benchmark an LLM on a single language, you'd get better results by training it on multiple languages rather than just the one. Scaling quality training data almost always increases performance as the neural network picks up general rules and an understanding of code that are language agnostic.