We have laws and oversight surrounding phone surveillance: https://en.wikipedia.org/wiki/Lawful_interception

Just like you say, it still gets abused both within and outside the government (eg. Salt Typhoon) because surveillance inherently threatens society.