Votes (particularly on submissions over comments) do not directly translate to karma. I'm not sure if it's documented anywhere what the algorithm is, but it's something proportionate to the logarithm of post votes becomes profile karma. It's similar with comments, but I believe (anecdata and observation) the positive effect of votes on karma is also logarithmic, but the negative effects of downvotes is linear; so if you have a highly controversial comment that sits at 1-2 points, you can net lose account karma.