You're not liable if the animal is just passing through but if you're feeding the wildlife (actively or just have a bunch of open trash cans) and the wildlife decides to do wildlife things to your guests, you certainly could be liable.

A hole in a fence is kind of a middle area. Did you create the hole for the raccoon to enter your property easier or for it to leave easier? Rabbits sometimes hang out in my backyard to eat grass and my dogs love to chase them. They usually enter and exit under the fence in the same spot (near my neighbor's vegetable garden) but that spot is near the backdoor and they are sometimes trapped if they are in the middle of the yard or further back (they always escape but there's been a few close calls). I made a small hole at the opposite end of the yard for them to escape easier. But if I knew a critter was struggling to get into my yard, I definitely wouldn't make it easier for them.