>averaging the last 50 reads and wait till the majority is either off or on.

This is a bad way to do it because it adds avoidable latency. A moving average is a low-pass filter. The switch bounce is better handled by hysteresis. Change state as soon as you see an edge, then ignore further edges until a timer expires, e.g. 5 ms, which should be enough for the bouncing to settle. A 5 ms timeout limits your repetition rate to 100 presses per second, which is beyond human capabilities.

You might want a tiny bit of hardware low-pass filtering too, for EMI resistance, but that's with microsecond-scale time constant, not milliseconds.