Alternately, it captures a bit of so many things (tech-debt in codebase, mental health of team, task risks) that it's best to avoid trying to link it to any one thing.

The past X weeks of point-estimates is what you use to forecast which things fit in the next Y weeks, and you can't have both stability and forecast accuracy. Any attempt to permanently "peg" a point to a certain number of man-hours is going to interfere with that accuracy.