I might be missing something, but a single 6 sided die can have all permutations for 3 players; a 24 sided die for 4 players; a 120 sided die has all permutations necessary for 5 players. Each one of these would even firmly support fewer players.

By the time you got to 120 sides you probably couldn’t label the sides with the order and would need a lookup table or something. That’s a disadvantage, I suppose.