My land is adjacent to public land and yes we have people coming through all the time. There is a forest road that becomes a private road and cuts through our land and people use it. We could put up a fence if we wanted but we don't want to... we're high enough up the hill that as long as people are being cool it's nbd. There are signs and most people (not all) are respectful.

I've found that living next to public land is an insanely undervalued luxury. There's very little I would trade it for.