You can either require mesh vpn or make it easy for friends to access your library. You can't have both.
I stay purely home network, as I suspect most do. So although I can see your use case, I can also understand why it's not a bigger priority than those other features.
Then whitelist their ASN.