I get the annoyance at "ask me again later", but "maybe later" or "not now" is helpful - there's no way to trigger dialogs by yourself, so this is a promise that even if you say "no" in the dialog, you can still go into settings to change your mind. Gives me less anxiety about the choice.
Then build better interfaces. Don't ask stupid or nefarious questions.
A much better way would be “Yes” and “No” with a small message of “you can change this later in the Settings”.
Or have three buttons: “Yes”, “No”, “Remind me Later”. Apple does stuff like that in other parts of the OS, like when asking for app permissions, so they clearly know how to do it.
How about a choice: both "no, maybe later" and "no, never"