Yeah, that's one of the points of the EU AI Act, I don't have the specifics, but for some critical sectors decision making cannot be automated by AI and you are obligated to get a human explanation if you request it.