> The thing is that's where most of the leaning and 'intelligence' is

The question is: Is it required for AGI that the model changes its weights _during deployment_, or can we train up and deploy like we do now and manage learning via context?

Taken to extreme, "context" could be defined as the "change in weights from training time" so the answer is trivially "yes", but that seems like cheating.