The usual way of solving this is to make the voucher responsible as well if any bad actor is banned. That adds a layer of stake in the game.
That's putting weight on the other end of the scale. Why would you want to stake your reputation on an internet stranger based on a few PRs?
A practical example of this can be seen in lobsters invite system, where if too many of the invitee accounts post spam, the inviter is also banned.