No, the dead can't exert such influence from the grave. You're dead. It's the next generation's turn. If you were the kind of person whose diary the kids or grandkids want to see thrown in the trash, then that's how it is. You had your time on the face of the earth, you don't get to haunt the descendants. You got erased, that's it. Or your story gets retold in filtered ways in the fog of the past.
Im perfectly content knowing just vague information about individual ancestors 3-4 levels up, and basically nothing on level 5, except the odd church record of births, marriages and deaths giving a rough indication of where some of them lived.
The people 5 generations back deserve much less thought space than the community you cultivate around yourself today (including living family).
This is frankly a selfish take. Society is a compact between the dead, the living and the yet to be born. Having a piece of someone’s deepest thoughts is a treasure for future generations. By deciding to destroy, you choose to destroy the past that the future generations might value.