>you shouldn't just leave to door open to potentially known bad users / traffic

Why not? If security is the priority, the worst that could happen from that is clutter in logs and higher resource use. Nobody wastes 0days spraying them on random hosts, and a real motivated attacker targeting your server will not be stopped by an ip blacklist.