Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.

Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.