I've had the same experience. Caffeine is super addicting, the ritual & habits surrounding it is a potent pull. For myself, it makes me erratic, impulsive, more reactive and agitated. One cup a day puts me on edge, makes me sweat more, makes me more intolerant, makes everything feel too slow. It such a sneaky drug and it can really get under your skin without you realizing how much it changes you.
I don't have the same experience, and I drink one cup of coffee (270 ml) almost every day. No agitation, no impulsiveness. I can drink coffee in late evening (let's say 8 pm) and sleep well. I guess I'm trying to say that we should not project our own experience on others, everyone is different.