The speed of air exchange through openings in your room (windows, gaps etc) is proportional to the difference of temperature between your room and outside. The more different the temperatures are, the more intensive convection is, and convection is very effective at moving/exchanging air. In summer the difference in temperatures of your room and the outdoor space is much smaller than in winter, which subdues convection, and the stale air tends to stay.