Do you honestly think that our society was more (or as) equal 200 years ago than it is today?
Or even 50 years ago? Even in the 1970s, there were places in the United States that women couldn't get a checking account without a man co-signing on the loan.
We can certainly take issue with how rich countries oppress and exploit poor countries today, but you can't honestly say it is worse today than it was during colonialism.
Prior to the enclosure period of English common law, land was assumed to be available to the private use of government subjects. The American West was the same before it was swallowed by the US government. The current situation is provably less free along this dimension. You could argue that this is not important relative to other things (this is a value judgement), or that things done outside of the state are outside of the discussion (this is a a fallacy, a lot of things once outside of the state find themselves inside it today), or you could say that this does not account for intra-community conflict between settlers and natives (this is a straw man)