Another reason I can think of is simple disagreement about facts (or, relatedly, disliking the lack of sources/evidence in the comment).

Like it or not, downvoting for mere disagreement is allowed here[0]. I personally think that any such downvotes should at least be accompanied by a reply, but I haven't been able to find official guidance about that.

[0]: https://news.ycombinator.com/item?id=16131314