You are missing a major driver of this. The main reason fertility rates have been decreasing is because of decreasing infant mortality. The number of children surviving to adulthood has been relatively stable until recently.