One thing I'm really scared of is EV charger software being modified by users, hackers or bugs to pull max power at times that don't suit the grid.
In the UK, for example 10 million EVs all pulling 7kw would overwhelm the roughly 70GW potential of the grid. Even a million EVs charging at an inconvenient time could add a 7GW draw which is enough cause a problem.
It will first damage the batteries very fast, second, most users don't want to mess with that, they want to plug and play. So, on both counts your fears are misplaced.