I’m inherently skeptical of big walls of text like this these days.

(So here’s a big wall of text of my own!)

However, a lot of what is written here makes sense.

And particularly “if your comprehension level stops [here] you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor”

This is exactly the problem.

And another point you make:

> but the sycophancy of modern day LLMs is an actual danger that enables more harm than good

I don’t think it is necessarily the sycophancy that is the biggest problem (though that is definitely a problem) but rather the combination of authoritative sounding text plus “complete answers” which sound wholly believable but are deeply flawed unless you have domain expertise.

I moderate a forum that deals with people who face a relatively common but somewhat complex (and nuanced) set of legal problems.

The purpose of the forum is peer support, shared experience (“lived experience”) and community.

It’s not legal advice, though moderators will sometimes step in to highlight relevant legal resources (e.g. case law/precedent or primary legislation/instruments).

Prior to AI infecting the forum someone would post their problem, people would respond with their often incomplete or poorly communicated thoughts, the OP would ask more questions - or argue - and a dialogue would occur. That created a community and people would post updates and ask more questions and find common shared experience. Many of them became correspondents with each other and some became actual friends.

In the past 12-18 months the discourse has changed from “here is my personal experience and here is what I did” to “here’s a bunch of stuff an AI says and I’m pretending it is me giving advice”.

Almost without exception the person who has started the thread will react positively to the AI generated content, even when it is egregiously incorrect - but won’t ask questions.

More problematically, these AI posters will often argue specific incontestable points of law “because I asked ChatGPT/Grok/Claude and it says this” and ChatGPT clearly cannot be wrong. And the border of precedence seems to be ChatGPT, Grok and then Claude some way behind.

I’m slowly seeing a pushback from people as “normies” begin to spot AI. But it’s ruined a community because the advice sounds so authoritative and complete that people won’t argue or ask questions.

As a result we have banned AI generated posts and remove repeat infringers.

That’s significantly reduced the volume of posting (below what it was pre-AI) but has significantly increased the value the members are getting.