I think the Luddites have a case because LLMs are irrelevant to the "search is your address bar" use case. Google took off when it became easier and faster than remembering the URL when you had a specific page in mind. The AI-summary contributes nothing to this except moving the link you want to click on several centimetres down the screen.
Those queries tend not to trigger the AI overviews, or they trigger below the first result and so do no harm if that's all the user wanted.
Luddites had a good political position, and their demands were reasonable. Of course, the capitalist classes used the power of the state to put them down with violence, because police exist to protect the property of the wealthy, not to actually support justice.
Maybe don't frame everyone whose not a FOMO AI booster as a Luddite? Some of us want balance and nuance, and to treat AI as normal technology. It's mildly insulting and if you know your history, inaccurate.