> This is insane.

As a human, this is fantastic.

> How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.

Personally, I hope it can't, or the fix is onerous and kneecaps a lot of the utility of AIs.

I wonder if we can even have some adversarial competition, where competing businesses try to get AIs to ingest bad data to pollute their competitors' decision-making.

For starters: lets get some fake math papers out there. Given proof-checking systems, maybe they won't totally stop OpenAI, but hopefully the bad math could at least drive up the cost and slow AI down.

True intelligence knows when it's presented with a proposition, whether or not the proposition contradicts one or more of its existing beliefs.

LLMs can be trained on arbitrary token strings, which semantically contradict each other, because semantics is not involved.

The way to attack the "garbage out" problem with an LLM (like anything else) is to address the "garbage in" problem.

You will get better quality out of an LLM that is trained on nothing but mountains of unvarnished truth: logically sound propositions and true facts about the world.

But even then, interpolating among such is not reliable! We can't add together two different truths and divide by two to obtain a midpoint truth: that makes no sense.

That is still a sort of "garbage in" problem. But that one is not in the data; it is in the design of the processing concept. The body of algorithms that power LLMs have a fundamental garbage aspect to them that cannot be removed; only a replacement of the LLM with something different can address it.

I've been thinking for awhile it is very strange that claude/ openai etc google "What was the inflation rate last quater" instead of just having a tool to query BLS datasets and that sort of thing.

So hopefully most of these statistics move towards first party sources.

> How can this be fixed?

You can find people complaining that Google search results decayed to being unusable prior to LLMs being a big thing. The marketers and spammers effectively won that fight. They'll presumably win this one too.

I don't think it is up to us at this point. LLMs are now the lens ("filter for what people see"). I hope, they will fix this. I think that they will still rely on reasoning and better grounding, which should prevent straight lies. But I think two bigger problems here: 1. training data might have the BS which would steer reasoning; 2. grounding might still be limited by small number of sources.

It cannot be fixed. The most hilarious example of it is YouTube. They have a video summarisation feature, which automatically creates video summaries that then get fed into the search engine. The problem with those is that they are created using models that are good few months behind, and let's say someone published a video on Nikon ZR and the model used to create summaries does not know about Nikon ZR and inserts references to Nikon Z8 or Nikon Zf into the summaries. This decreases accuracy of YouTube search and negatively affect discoverability. It's the AI version of shitting into the well you drink from. Bon appetit!

>How can this be fixed?

People stop trusting what they read on the internet.

All alternates are just attempting to improve the baseline trustworthiness of random information from the internet (and/or LLM that's baby-birding it back to you). Any increase in how much people blindly trust random stuff from the internet only serves to increase the blast radius when it is wrong.

The fix that seems obvious to me is to only let the LLM see truthful (or at least non-malicious) data. The model trainers carefully curate the training data, and you don't give the LLM access to full-web search. Instead, you hook them up to an 'official' source like an encyclopedia, documentation for the software in question, etc.

It's a ton of effort and expense, but probably feasible from a technical point of view. So instead, the model providers will probably do something silly like wire up rules like "NYT=Reliable;my.blog.forbes.com=Unreliable" and they'll say the issue is solved.

>How can this be fixed?

Defer to authority and trust your government to rank what is truth and what is not, so your LLM always gives you the right answer. Nothing can go wrong here ever.