"Hardiness zones in this map are based on the average annual extreme minimum temperature during a 30-year period in the past" https://planthardiness.ars.usda.gov/pages/how-to-use-the-map... (the page also reflects this in one of the sections).

So it doesn't matter if it was -20 F all year or for 1 day, either counts as 1 value of -20 F out of 30 annual measurements to be averaged. 2 record breaking cold winters out of 11 wouldn't necessarily imply much with over 1/3 of the averaged data being replaced (it'd depend more on the 11 years replaced than the 2 new extremes), and that should show in the graph of the exact yearly values used towards the middle.

I do sort of wish it opened with the deeper explanation of what the data is first and then got to showing you the values second. Giving someone numbers, having them try to interpret them, and then telling them what it was the numbers were actually trying to measure is a recipe for creating confusion.