alt Hacker NewsPublish fake stats, get cited in LLMs
Comments
I'm reminded of a section of Cory Doctorow's latest non-fiction, The Reverse Centaur's Guide To Life After AI (it is of course an excellent read), where he discusses how the semantic web and having an internet set-up for agents and AI to use won't work:
>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.
There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.
>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.
This problem is so bad that you can create a website just to review your own website so when LLMs want to pull out reviews for the website the fake reviews count
This has existed for a long time before LLM, although in a slower form. On Wikipedia it's called "citogenesis".
This is insane.
How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Of course, LLMs already use Reddit, Facebook and co. as a source, so it doesn't really matter...
LLMs are facing the same problem pre-google humans faced:
Is this information on this website true/high quality?
We need a google for LLMs!
Oh, but LLMs don't consume ads.
We need ads for LLMs!
Help.
It is particularly bad when trying to find the reality behind common misconceptions.
Even when there is no evidence to support something, and no motivation for anyone to promote the falsehood, models can build an ediface out of false claims.
There are always people prepared to invent a story to explain why something is true. Just so they can sound knowledgeable. They aren't trying to perpetuate a lie, they are just under the same misconception as others, quite a lot of them will believe their invented explanation even as they say it.
It's going to be a difficult task making an AI pick the correct one of two mutually exclusive pieces of evidence, especially after it has found one piece and does not yet know the other exists.
See Chavda’s paradox https://zencapital.substack.com/p/chavdas-paradox
What would little Bobby Tables do?
Happens 98% of the time
This is fine work, we need more communication to the general public about this
Trust isn't exactly a new problem in ai. Guess why the semantic web never had a chance? There's more incentive for fake than for fact.
> In cartography, a trap street is a fictitious entry in the form of a misrepresented street on a map, often outside the area the map nominally covers, for the purpose of "trapping" plagiarists of the map who, if caught, would be unable to explain the inclusion of the "trap street" on their map as innocent.
This is a known problem, when ChatGPT uses web search tool, top 10 results are often LLM generated sources as well.
They now train on their own content.
Eating your own d̶o̶g̶ f̶o̶o̶d̶ poo.
How surprisingly full-circle; AI is really becoming indistinguishable from humans! If you think about it, AI searching the internet is just like a human scrolling twitter. Maybe it's entertaining for them :)
Another fun one, post on HN, wind up on Google Instant Answers, and quoted by Gemini (or whatever the model there is). There's been a few times I've googled something insanely niche, clicked through only to find my own comment as the source, turns out my mind runs in cycles where I randomly ask questions I've formerly answered, and now I can answer them for future me.
It's both a blessing and a curse, especially if I'm wrong. Sometimes the Google AI misappropriates someone else'se quotes to me, not sure why, though those are usually if I ask it about myself directly and it tries to generate a summary of my HN post history.
On one other occasion... it was a post I had posted 2 weeks prior, I was trying to do further research into the subject.
I wonder why did Google derank it. Did it correctly identify it as LLM written and sink it because of that?
this is actually insane. and people still use chatgpt to generate reports for their managers. snowball of fake data
I suspect there are already many fake stats sites like this by the time I read this.
Yes, this can be abused to rewrite history too.
Related: Preserving Historical Truth: Detecting Historical Revisionism in Large Language Models
> Its made-up numbers have also ended up in an academic paper and in a bank's investment research.
It's an excellent way of showing how unreliable the information you find online is (and it will only get worse in the future), but willfully creating misinformation and made up shit and offering it as fact feels hostile to humanity.
Surely nobody got hurt as a consequence of those "facts", but bad decisions with real world impact could be made regardless. It feels like the behavior should be shunned in a general case (the same way I'd shun those numerous spam sites that pollute search results and very much would prefer they not be allowed to exist), even if in this case the intent was good.
Problem is, you can't really do anything about it, other than figuring out a way to take down hostile sites and keep them from popping up, so it's even worse than fraud or even just bad research in academia.
If I'm not wrong ChatGPT indexes on Bing, curious if the site and stats rank well on Bing
A little off-topic but I have been thinking about how writing changes in the age of AI.
I wonder if similar to how many engineers now write 90% or maybe 99% of code with AI and primarily weigh in on the plan/design, will writers too primarily weigh in only on the content brief/plan and then the actual post is seldom read in full (once you have faith that the AI is not making up citations). Because the audience of most writing is now bots prose and style are no longer relevant.
"You really think someone would do that? Just go on the internet and tell lies?"
Congratulations. You've invented the International Business Times, The Conversation, Forbes "blogs" and any number of low-end and mysterious outlets that exist to be cited by semi-serious outlets like Politico, Wired, etc.. Then, those get cited by television news and NYT and WaPo in quotes for the first article, and without quotes in all subsequent articles. Finally, those new "facts" become enshrined in Wikipedia and come up at the head of every search for any related terms.
A lot of "facts" get created this way. You end up with hundreds of citations, but nobody seems to know (or care about) the original source. And meanwhile, freelance militant conformists (rarely shills, usually people intensely addicted to updoots) use those citations to personally attack people who cite primary sources, and to get those sources banned if possible.
edit: Shills are ancient technology. All you need now a fleet of repost accounts that exist to updoot fools. The fools, having the happiest moment of their lives, then double and triple down, becoming more and more gleefully hostile. That's when you finally have to use a shill to get them to open a patreon. Then you can get the world out of them for a couple grand a month.
I thought everybody knew this was a problem. Scrapers vacuum up data but don't have any way tell what's true or false.
It's interesting to have a specific case where obviously false data is being spread, but there's nearly an infinite number of more subtle variations around.
Nine out of ten stats are made up on the spot.
don’t
if you just lie and an llm or a newspaper or a kid believe you, at that point it isn’t a criticism of their naivete, you are just lying
similarly hacking through systems or picking locks isn’t a showcase of weakness if systems, it’s just being a hacker
[flagged]
[flagged]
[flagged]
The company I work for disappeared from Google as well and it is totally a legit business.
After dozens of requests to check we finally received a message saying they were going to roll back changes.
Well. We received that at the same time they published a post saying they would do that.