From what I have seen, layoffs tend to go hand in hand with pushing for more work from the office (which companies hope will help them get rid of some employees without severance), so if being in the office helps combat layoff paranoia and depression, there's that.
I personally don't have less paranoia now I am in the office 3 days a week (and probably won't have less when I will soon be forced to come in 5 days a week either), but I only talk to "my" team via Teams no matter if I'm at home or in the office, because they're spread over several locations (and no, it doesn't make sense to do RTO in these conditions, but that just reinforces my certainty that all the talk about "increasing efficiency" is just BS and they simply hope to get rid of people, no matter how competent they are). In the meantime, I don't really care one way or the other, but if they want to get rid of me, they should at least offer me some money for it, I won't do them the favor of just leaving voluntarily (especially not in this economy).
You say your team is remote "anyway", but RTO precisely is about connecting with other people OUTSIDE your team, working on internal politics, and you act as an interconnection point between the remote team and the local network.