Hearsay has it that the reason is spam. Spam messages are said to have massively higher chances of minor RFC violations when they arrive at the destination server.
Most of the time, in my experience, when one encounters a situation like this in Internet tech (i.e. "why is this suggestion treated like a hard requirement?"), this is the answer: "because attackers found a way to exploit the lack of the suggestion's implementation in the wild, so it is now a hard requirement."
The standards, to my observation, tend to lag the CVEs.
Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.
Most of the time, in my experience, when one encounters a situation like this in Internet tech (i.e. "why is this suggestion treated like a hard requirement?"), this is the answer: "because attackers found a way to exploit the lack of the suggestion's implementation in the wild, so it is now a hard requirement."
The standards, to my observation, tend to lag the CVEs.
Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.