Speed limit is only a proxy for your braking distance in case of emergency braking (at least in most cases, but also a proxy for bad road conditions in others): the point is to ensure safety, and not a particular speed.

I've driven behind drivers driving 25km/h in a 40km/h area and not stopping for pedestrians at a crosswalk with right of way (if somebody jumped out elsewhere, they'd probably just run them over at 25 km/h), whereas I always do even if I am driving at 45km/h because my foot would be hanging over the brake near areas of low visibility (like intersections) or near crosswalks or with pedestrians near the road.

Your braking distance is largely a function of your reaction time (attention + pre-prep + reflexes), and your car performance (tyres, brakes) on top of the speed, and speed limits are designed for the less than median "driver". You obviously have most of those under your control, but the speed is the easiest to measure externally.

The obvious counter is that I could be even safer if I also drove at 25 km/h, but it would take me much longer, I'd hit many more red lights, so I might stop being so attentive because I am going "so slow" and taking so long (maintaining focus is hard the longer you need to do it).

However, measuring individual performance is prohibitively expensive if not impossible (as it also fluctuates for the same person, but also road and car conditions), so we use a proxy like speed limit that is easy to measure.