Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase.

I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.

What's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state...

For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff

Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.

Yes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers.

The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.

> Is anthropomorphizing a real problem?

The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well.

For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc.

Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.

> Is anthropomorphizing a real problem

Yes. There's a difference between scrapping a session and starting over, or going back and branching something, or using sub-agents to see five outcomes, vs arguing with a system in a long drawn out chat.

Like - I know that if a model starts doing something silly, instead of correcting it - I can probably go back and edit two steps prior to add an extra guardrail, or extra data, or whatever.

> Is anthropomorphizing a real problem?

Yes it's really a problem. On this website you are surrounded by people who have technical knowledge and understand at least somewhat, how a computer functions. You have the ability to separate "fun" and "reality" because you know you're putting input into a really really big calculator. Most people do not fathom this.

AI Psychosis is a real thing, look it up (don't just ask an LLM) and do some reading. It's actively harming people, and the way they think. There's no regulation around any of this stuff and it drives me crazy that we let these AI companies _sprint_ so far ahead of everyone, and now we're facing the consequences.

Simplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.

> Is anthropomorphizing a real problem?

Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.

> It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.

A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing.

Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.

> none of the serious LLM researchers believe it has anything to do with human reasoning

But some of the biggest evangelists, who are well respected programmers that get lauded on this very site, have said it is fully sentient and has emotions. Even going back to 2022, when the LLMs were dogshit, a Google employee lost his job claiming it was sentient because it said it had emotions.

Combine that with the marketing angle of both Anthropic and OpenAI, who have been trying their hardest to describe every function of an LLM as analogous to the human brain. Because it's politically useful to paint them as dangerous and uncontrollable, so the keys will only be granted to the few people on the mountaintop.

I find it annoying because when I read ML papers nowadays I have to back-translate from anthropomorphized talk into actual machine talk, then mentally compare to what I actually know about brains and cognition.