It's a nice idea, but couldn't it also be explained easily by the invaders enslaving or killing most of the population? Existing structural decline and instability makes this a lot easier to do for an invading force.