I've done a lot of exploratory work with Stable Diffusion LoRAs, and I actually do buy that there's some juice here, though it's almost certainly not nearly as good as other techniques can be. In particular, this technique will likely avoid the intruder dimension problem which plagues naive LoRA. SVD is expensive, but you only have to do it once at the beginning of training.

I haven't done much research lately, but when I was working on it, I was having substantial success training an adapter of the form U_k @ P @ A, where U_k was the top k left singular vectors of the underlying weight, and then P and A were your typical LoRA projection matrices.

The 13 parameters are kind of misleading here; the real juice is going to be in the P_i fixed random matrices. My suspicion is that they are overfitting to the benchmark, but they almost certainly are observing a real gain in model capacity that is largely due to avoiding the intruder dimension problem.