If you break down the stats, a significant fraction (if not most) of the fatal accidents in GA are in experimental aircraft like kit builds and exotic stuff like fighter jets, despite accounting for a small fraction of the total flight hours. Most of the rest are due to preventable human error like misjudging your ability to fly in bad weather or miscalculating fuel and running out mid-flight.
It’s like driving: it’s usually very safe if you don’t speed and drive tired or drunk.
With the distinct difference that driving isn’t at all like flying for very obvious reasons.
Miscalculating range in a ground vehicle typically has the somewhat less than fatal outcome of being at least somewhat embarrassing.
Misjudging driving ability is largely a characteristic attributed to male drivers under the age of 25, and the overwhelming majority of incidents are more costly and ego injuring than fatal.
Last time I checked, the average driver can expect to be involved in a fatal car accident every 200 million kilometres or so (Australian data).
While general aviation appears to have a fatality rate of around 10 per million flight hours.
Average speed in a car is typically well under 100 kilometres per hour, making general aviation fatality rate 10 to 20 times higher.
Having said that, the law of small numbers informs that the average general aviation pilot can expect to be involved in a fatal incident approximately never.