I worked on Windows apps for many years. The problem is that the Win32 interface only really gets you 90% control. The hardest thing I've ever done was when our product manager decided that color theming our app was an essential new feature. That's when you find out that certain Windows features like scroll bars under certain conditions bypass the usual message loop completely, because they know they can get away with it. The part I remember most vividly was recreating the lowly MessageBox from scratch, because none of the internals of the system supplied one were exposed in a way that you could modify them.
You can modify this stuff if you go deep enough and are willing to detour the native Win32 API functions. Some things implemented in User32.dll don't make the appropriate API calls back to other User32.dll functions, and you need to detour Win32U.dll instead.
Wow this could be painful, to try to recreate something from scratch. Would QT make it work?
My memory from the old days is you can use Win32 hooks to modify the MessageBox. HCBT_CREATEWND gets you the HWND of the MessageBox, and you can subclass it (in the Win32 sense) to insert your own WndProc. Then you're off to the races--it's your dialog now.