You should care how much maths and encryption you use [0][1], because this is not only possible, but there are multiple approaches.
[0] https://satoss.uni.lu/members/jun/papers/CSR13.pdf
[1] https://fc16.ifca.ai/voting/papers/ABBT16.pdf
More important than lack of voter fraud is proving to the population a lack of voter fraud.
More important than lack of voter fraud is proving to the population a lack of voter fraud.