Jim Roskind had a really nice talk [https://www.youtube.com/watch?v=_uaaCiyJCFA] at AWS Reinvent 2022 which highlighted the fence post problem. Basically, AWS was targeting P50 and P90 latencies in many services, and sure enough there were big clumps of latencies right under those target fence posts as engineers were basically gaming the metrics and nerfing the >P90 latencies to make it happen.