I went to a talk by an author of one of these papers (David Rand). IIRC the format of the paper's experiment was collecting people with conspiracy theory beliefs and having LLMs with different strategies try to reduce their confidence in the conspiracy theory.

The best strategy, perhaps reassuringly, was tailoring facts to the person's worldview. I don't mean massaging facts, I mean that if the person thinks 9/11 was an inside job because of something about melting steel, then the LLM will present some sourced facts about how that actually works. And then it will do the same thing for all six of the sub-beliefs that support their overall belief. This was key to the strategy: the LLM has enough breadth and recall to exactly and patiently rebut most mistakes a person is making without making them defensive.

This strategy handily beat other things like flattery, selectively lying, etc. So while LLMs can clearly persuade, the evidence suggests it is easiest for them to persuade belief in something that is actually true.

This was in 2024 or 2025. Things might be different now.