To each his own, but also:

All things in moderation; I drink a cup or two a day, but only before lunch, never after, and I eat beforehand.

If I don't eat, or I have too many coffees together, I get the anxiety you mention.

If I have coffee after lunch, it affects my sleep.

But, accounting for those things and mitigating them, I now not only get the benefit of coffee (if there is one), I get the social benefit of having coffee with people.