Most astronauts were chosen from a decent sized pool of military pilots. Pilots are some of the most expensive assets the military has (moreso than the planes they fly) and they have to be physically fit. People wanting to become astronauts are subjected to rigorous physical testing.

No, they're not Olympic athletes but they're considerably more fit than the average American.