Because a higher percentage of calls from Google Voice are fraud. It's really that simple, they don't care about you, all they care about is how well each available signal correlates with fraud percentage.