I think you need to be at a certain level of karma to 'earn' the right to downvote.

Flag is for inappropriate content / behaviour.