I think that's a problem domain in itself. Identify when failures that occur even when skilled people doing their jobs properly, adjust the environment so that they no longer happen.

It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.

Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.

Consider one of the problems listed in the article

>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".

The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.

It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.