you don't need an agent for hsm.

also, don't know a good security researcher who uses agents (some even use ssh config to limit which keys are even tried per domain)

ssh config hacks are also a must if you have, say, two or more identities to github.com for example.