I did the same a few years ago. I can't think of a single time where I wish I were still on Prime. In the rare instance where an item isn't eligible for the $35+ free shipping, I switched my order to Walmart or Best Buy where it was.
That makes me think, why do I always buy from Amazon before other sellers when prices are the same? It's definitely Amazon's UX. Its fairly fast and low friction. I only pull away when I can save money elsewhere.