The "something deeply wrong" part about AI, that even most technology enthusiasts evidently do not seem to grasp, is that it is still fundamentally a statistical model — an algorithmic construct — and does not possess any real intelligence or critical thought whatsoever.

No matter how much investors and tech companies want you to believe that they are on the verge of super intelligence, nothing I've seen to date can not easily be explained by "correlation engine", including the "novel" math solutions, all of which appear to just be "a composition of solutions humans have developed and documented elsewhere" upon deeper inspection.

When I read AI-generated prose that is aimed at the general public, I have the exact same feeling.

But when I ask Codex a technical question about coding, I don't get it at all. Codex replies to me in a very direct, technical manner, similar to the way I speak.

When I ask ChatGPT to be concise and technical, I get the same effect.

I think it's because prose aimed at the general public has to be very attention-baity --like the textual equivalent of a Mr. Beast video--, not because AI is incapable of writing like a human.

Completely agree. AI is very impressive in many ways but there is something deeply wrong that is hard to put into words. The output is probable but never true, if that makes sense.

I think this is also the mechanism behind why AI generated videos and images are so captivating at first. I remember when Midjourney first launched and it was hours and hours of a brain-melting "Wooooooow". But once you get used to it and start to identify the patterns the brain quickly labels most AI-generated content as blank space.

If the image or text wasn't created by a human, then there was no intent behind the content, there is no message or novel information conveyed, and it reads as noise.

You are re-compressing information that is in-effect meaningless because it's all decompression artifacts.

The AI had a nugget of data and decompressed that into a flood of text.

The exhausting thing is that we're then trying to re-compress that or derive the original intent and meaning from noisy decompression.

It's like un-zipping a zip file into a probability space of what could have been in the zip -- and then having to find the actual files worth reading.

For some research I looked up some very old Reddit threads a couple of days ago.

And, Oh my god, you can actually see how this style of writing influenced AI writing today, I constantly had to remind myself: "this was posted before ChatGPT released".

The reddit influence is especially true for "storytelling" writing.

Yeah, I have the same problem. There's a good quote example of this:

> There’s a growing scissor between people who are happy to read AI and those who violently bounce off from it.

> People adapt in different ways — and some people absolutely cannot look at it. That cognitive split creates a surprisingly powerful opportunity: you can write something that, technically, sits right there on the page, yet an entire sub-population will be incapable of staying with it long enough to actually read it. You can hide entire sub-structures in plain sight. It’s not avoidance — it’s adaptive obfuscation.

> The paragraph before this one was the only thing generated in this essay and if you just skipped over it I highly recommend reading and really understanding what it’s saying.

It's quite effective. I think this kind of text functions like the chumboxes you see at the bottom. Taboola and so on. Just mental ad-block takes over.

Do you have much exposure to pre-AI corporate memos, mission statements, marketing plans, or white papers? Because they were mostly written in that style. Full of buzzwords, cliche similes, platitudes, jargon and stock phrases.

> just short-circuits to "there is no information here"

That is my experience with the way the models write by default, often even when instructed not to do that. With enough effort you can get even them to slightly unslop the writing so it doesn't read like some LinkedIn/Buzzfeed brainrot, but the problem is that it's not trivial to do and most people won't do it, so the default is indeed horrible.

People should notice that it is constantly inventing plausible jargon, some of which may or may not have been used in some specific context.

I kind of wonder if our ability to skim has been stymied.

blah blah blah

- blah blah nugget blah blah

- blah blah blah wrong blah blah nonsense

- blah blah blah obvious blah blah

- blah blah blah off-base

blah blah blah

It is that we HAVE to skim because the text is so cheap, and it wears us out.

This is just a weird feeling that I've been coming closer to articulating lately, but I only think that you can get forward reasoning from what is basically word association; there's no mechanism for unwinding it because it has no real memory. By "it" I mean word association itself, not any context window. It predicts what could be in a position, and ignores what wasn't in a position.

People don't do that. People are constantly engaging with paths not chosen. Right after I choose to write one thing, I'm immediately engaging with what I chose not to write there - I'm explaining why I didn't write it, I'm realizing that my choice may seem unusual so I'm trying to make it memorable, I'm focusing on the distinctions between what I wrote and what I didn't.

LLMs don't currently do that. LLMs just ape a structure. When the structure resembles the sort of timid, clarifying fussing I just described, the LLMs just drift randomly because what they didn't say wasn't in the context.

I also think that's why they have such a serious problem backtracking. They're not taking into account the already eliminated possibilities. Often the thing that was so unlikely that you weren't going to waste time on it is the answer, and things you discover while going down an ultimately wrong (but initially far more promising) path remind you of the path not taken.

They're simply assembling a thing that resembles a valid argument, and happen to make sound choices because the plurality of input happened to contain sound choices. This is usually a very good bet because there are so many more ways to be wrong than to be right. But it doesn't account for attractive (common) wrong choices. You need a way to back out of those.

It reads like the white papers companies publish on their websites to build legitimacy. Or anything from those IBM / SAP / Deloitte / etc consultants who write technical papers despite having little to know understanding of the technology.

That's why the business and government people love it, they spend their entire careers reading this nonsense.

> my brain immediately recognizes AI generated text

I bet it does. I bet it also recognizes some human text as AI text, and doesn't detect other AI text.

Are you sure you are not doing the same thing with other texts?

I started to skim a lot more text due to me having read a lot. Like in news article, i stoped reading the first paragraph because it repeats just what it was already written in the short subtext. Then there is the second paragarph which is used to have some historical view or whatever it is.

The junior engineers at my job have a terrible problem of writing AI "proposals" to problems. The proposals are all extremely detailed and verbose to a thought-terminating extent. It takes a lot of effort and self-control to parse out the actual "ideas".

I think of the Dwight Eisenhower quote: "Plans are useless. Planning is indispensable."

The process of thinking through a system and communicating your design to other humans is a core part of software engineering. You want to build the right abstractions and communicate the right level of detail. Delegating all that thought to an LLM means your proposal isn't clear to the target audience, and it's not helping the author to understand the problem.

yes, but now I’m also experiencing that for human-written text

> There's some psychological mechanism by which my brain immediately recognizes AI generated text and just short-circuits to "there is no information here".

I think you need to self-correct here, because otherwise you'll be ineffective in an information setting, where I expect AI-generated resources will not only be the norm, they will absolutely swamp the environment.

It's like if on any website you went to you saw a lot of posts written by the same guy over and over again. Even if he used different names, you'd start to recognize him eventually because of his style. Seeing as he doesn't say a lot of valuable stuff, you'd also learn to skip whatever he says.

I do worry that it's just survivorship bias and we're also consuming higher-quality AI output that's indistinguishable from human writing, but we focus on the raw, unedited, low-effort AI slop and think that we're good at recognizing AI text. Even if we really are at the moment, it might not be long until AI companies figure it out. I'm not sure why they haven't yet, given how many books they've burned for this already. Maybe it's just more efficient for the model to stick to a single way of writing, I don't know. But when that point comes, we'll be back to the usual way of reading and interpreting text because there would be no way to tell what produced it.

we are working on it, the thousands of gig workers tuning frontier models

> Something is deeply wrong with AI generated output

It works just fine for me.

Yep. It's like it's painful to read for me. It's because the next-token predictor is just mashing (mostly) grammatically-correct and plausible sentences together, without any real intention or meaning. So everything sounds plausible, but almost entirely void of meaning.

Exactly, AI-generated text reads so smoothly, that the same short-circuit shifts my attention away from deep focus and onto scanning of the text, looking ahead to get the gist of it. Forcing myself to read the text fully feels almost painful. It's like reading a terms-of-service or any boilerplate document.

Once you see past the illusion I think there’s no going back. AI writing style is just dogshit. This hype wave is based on the belief that we’re inching closer to AGI but seems to me we just increasingly struggle to define intelligence. LLMs seem smart because they can pump out thousands of LOC quickly, and enthral you with fancy words and bullet points. I don’t fall for the intelligence illusion anymore.

I've got a 3 step instruction to compress Ai text into useful info.

1. Ask it to write according to the Google Developer Documentation guidelines. Gets rid of fluff, less emotional statements, no it's not x it's why.

2. Tell it you have extreme ADHD and need everything condensed as much as possible. You can always ask for expansion on an answer later.

3. Bullet points whenever possible.