I really only started drinking coffee at my first real job after grad school. They had free coffee in the kitchen, so I'd occassionally have one. Maybe once or twice a week. I was like that for several years, and would occassionally go weeks without a coffee. During that time, I was very productive and went from being a new grad to leading the entire team of veterans in less than five years.

After leaving that job, I now consume fairly regularly (for the past decade at least). I can still easily skip days without coffee, though I do prefer having it daily. I literally see no difference in my day to day between having coffee and not throughout my two decades of experience with coffee. I can just as easily fall asleep after a coffee and I rarely feel amped up from a coffee (if I do, then I just stop drinking it). I've certainly never felt anhedonia like many others have mentioned in the comments when I've taken breaks from coffee.

I think it's clear that people have different experiences with substances. Whether mine is a common one or not, I cannot say. But I do have a baseline to compare to and I can legitimately say the only thing that has ever caused me anhedonia was burning out from too much work (during burn out I was still consuming coffee and it didn't improve my mental state at all).

My prior would rather be that people have wildly varying sensitivity to caffeine genetically. Some get panic attacks, some don't feel much.

Only true if they drank a cup of coffee since their earliest memories began.