I... What?
This is literally the definition of selfish? You see what you must give up for the sake of someone else (children), see the lack of support you will receive, and decide that you don't want to make that exchange.
That's literally a selfish decision, because you are deciding you want to keep that energy and those resources for yourself.
It's not inherently bad to make that decision, but it absolutely is selfish.