"Our society only works thanks to most people being slaves to these tremors."

If everyone defects, the system breaks down. Morality is good and it is actually logical - it solves the prisoners dilemma and pushes cooperation instead of defection. It also reduces harm and has lots of other good properties. I feel that how we affect others matters, but even if you're just a sociopath doing the math, defecting is a strategy that burns things down at scale, not a smart one. Tit for tat with forgiveness is not only morally aligned, but also more prosperous in scenarios that aren't just one-off interactions with strangers.