I think maybe both of you are correct, in a sense - the population growth provides means (labor, competition, consumption) for increasingly extractive and unequal society.
You figure our society is "increasingly extractive and unequal"? That may be true relative to post-WW2 America, but on a larger scale--either historically or geographically--can you really hold that modern society is more extractive and unequal than slave societies, serfdoms, and empires of the past?
Honestly, population growth has, if anything, correlated to increasingly equal and egalitarian societies.
Innovation relies on bigger groups, which then allows a smaller portion of the population to do more.