This is called an oomkiller. The kernel has one but it kicks in very late and the kernel prefers to do page trashing instead of killing processes.
systemd-oomd should be integrated in systemd, you can configure it to your liking and see if it improves your problem.
I believe that you mean: https://en.wikipedia.org/wiki/Thrashing_(computer_science)
Chris Siebenmann discusses when the OOM killer triggers: https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen
Chris disables systemd-oomd after it obliterates his X session with no explanation: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...