People hate Congress but love their Congressperson. It's even got a name: Fenno's paradox: https://en.wikipedia.org/wiki/Fenno%27s_paradox
Diffuse organizations like "Congress" or "the party" aren't what you're voting for. Unfortunately Congress really does take it on the chin for the actions of the President in midterms, because that's very recognizable and personalized, and partisan votes (or just staying home) are the only lever people have to express displeasure.
"People hate Congress but love their Congressperson. "
I honestly think most Congress people want to do good for the country. But the system has become totally dysfunctional making it basically impossible to really work on issues. The influence of money is just too much. To me this goes hand in hand with the rising inequality that gives a few people at the top outsized influence. Super PACs don't even have to deploy their money anymore. The simple threat is already enough to bring most Congress people in line if they want to keep their job.