I agree with you that this is a noticeable effect, but I have a hard time figuring out why it is, especially on Macs with the sealed system volume (which should make the operating system files bit-for-bit identical between a freshly set up Mac and one you've been using for years). Certainly it could come from the accumulation of cruft in /Library and ~/Library, but you can clean that up too.