If your neighbor is a defector (a game theoretically optimal name for "an asshole") there's nothing to do but turn the other way and talk to your neighbor over the other fence.

Social exclusion, like putting a kid in timeout.

Maybe they will get the hint, maybe not.