I have almost exactly the opposite reaction. By not caring so much about the minute details of physical things, or having the very best croissants or whatever, frees you up to enjoy anything or focus on interactions with people, ideas, anything else.
Being able to enjoy/tolerate a cup of coffee from my cheap machine at home saves me €2 and 30 minutes of my day. I’m happy that I am not a connoisseur.
There’s a difference between caring about something and being able to appreciate it. You can do the latter without the former.