How would you design such a system to be granular enough to solve for arbitrary application needs without being so complex as to be impossible to tune without false positives and false negatives all over the place?

It's not like people have tried to improve core security models, but in my opinion it's not really compatible with the core filesystem abstraction that programmers and power users of desktop computers demand. I think you need to move to a completely different model—like iOS for example—to meaningfully improve security controls without nerfing the OS.