Who makes your ID card, generates the private key, and stores it there? Do you trust them? Did they retain a copy of the key? Did they implement security effectively (a very difficult task, considering the resources of attackers)? What about securing the manufacturing process?
In the end, you always must trust humans.
Indeed. The rubber hose always beats cryptography. Always.
So if someone else knows my private key and voted for me not the same way i intended to vote, this will be visible in final results. And then you can start asking questions. If they have my private key and voted the way i intended, then its no problem :)
This is one big advantage of paper vote systems. You can still bribe, trick or coerce a single person. But it becomes much more difficult to do so en masse. Computer voting allows fraud at scale. It is much more difficult to scale an attack on a paper voting system.