I agree that compromised source dependencies are the bigger problem, but that doesn't mean a compromised build infrastructure isn't. Just this last week, we had two Linux kernel LPEs that could have been leveraged to implement just such an attack, for example.

Another thing to consider is that Debian has quite a few derivatives who may also rebuild packages from source, so you have a multiplier there.