Now I think about it, it's kinda weird if non-root users can cause kernel modules to get loaded, without any hardware changes having happened.

If the kernel modules for esp4, esp6 and rxrpc aren't loaded - how is it that a non-root attacker can cause them to get loaded?