I found the input latency to be much worse than the PC connected to the same TV, for whatever reason.

I thought I surely was imagining it, so I revisited it 6 months or a year later, and, yes, the input latency from the controller is substantially worse on the Apple TV.