It always seems like a missed opportunity that trackballs don't detect when you twist the ball. It would be so natural for spinning a model in CAD or zooming.
The easiest approach would probably be to add a second optical sensor and do some math to calculate the twist rate.
Twist scroll requires two sensors and is patented by Kensington.
This guy made exactly what you describe
The Kensington SlimBlade scrolls when you rotate the trackball.
These have been made on and off since 1995 at the latest. They never work out even though you'd think they'd be super popular for 3d modelers at least. I first encountered one for SGI workstations.
https://en.wikipedia.org/wiki/SpaceOrb_360 as one example.