Not a holistic answer but one example: There are specific fine-tuning techniques like Orthogonal Subspace Finetuning (https://huggingface.co/docs/peft/package_reference/osf) that target 'unused' parts of the weights so you can keep highly informative parts of the weights as-is.