RAG Is Simpler Than You Think

I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search. Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.

RAG is basically good old information retrieval with LLMs doing the querying. This can include vector search but it works without that as well. Treating vector search as magic pixie dust that makes search great without effort is not necessarily going to work that well. Also, it can add a lot of cost and complexity to the equation. And if not tuned properly, you don't necessarily get good results.

The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).

With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.

The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.

More LLM-generated text about LLMs.

Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.

I have a particular antipathy for articles too lazy to spell out acronyms on first use.

So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation

I have built systems using all of these approaches (all in tandem). For the most part, the juice is not worth the squeeze (in building a highly optimised corpus-specific information retrieval strategy) outside of a very few fringe cases. The amount of technical discussion far outstrips the use case for RAG.

Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.

Then, throw it all in BigQuery. Handles all the vector stuff natively.

Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.

I assume other vendors than Google have a similar batteries-included approach you can just plug in.

I notice a lot of these AI written articles share this pattern where they'll present idea 1, then idea 2, and finally idea 3 which is some amalgamation of idea 1 and 2. Claude especially will present hybrid options and compromises to avoid having to make a choice then framing the hybrid option as the "best of both worlds" when they're borderline nonsensical.

"on the fly embedding" and "Sparse + dense reranking" don't really make sense how they're presented and smell like they came from a long claude-driven conversation after multiple cycles of these hybrid compromises across many turns.

There have been many blogs like this over the last years.

Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.

90% of "document" based RAG projects should view semantic search with embeddings as their primary method.

It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.

Agentic query rewrite on top of good old fashioned Lucene is the end game. This is effectively providing a lot of the same magic you get with the semantic approach. Allowing the agent to query the document store iteratively is where the capabilities become unbounded.

Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.

If someone has a Postgres db and want very simple RAG:

https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )

The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.

RAG stands for Retrieval Augmented Generation. The purpose is to search a corpus of text by meaning rather than exact match.

I had to look it up.

We removed retrieval from our coding agent a while back. What convinced us wasn’t a benchmark, we found that the retrieval path had been returning zero results for quite some time because of a technical bug, still nobody noticed, indeed it was working better than before.

After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.

Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.

(I work on TheGitAI, for disclosure.)

Chunk size matters way more than the retrieval model in my experience. Get that wrong and nothing else helps.

Althought I agree with the first point of the author that FTS is underrated in this new RAG-first framework, the whole article really hides all the problems with RAG-pipeline and kind of hand wave everything.

If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/

Note: I am not the author, just got her article in my newsletter and found it useful.

If like me you run models locally, it's pretty easy to run your own RAG locally also using a Vector Database like Qdrant for persistence, and a middle-layer like Mem0 for realtime retrial and updates. I documented the set-up steps here: https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Loca...

I would like to see how each recipe performs against its corresponding evals. Some sort of ranking would be useful.

Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.

> Why this is more flexible than embeddings

Oh boy...

Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?

I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.

I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."

Any info or advice would be appreciated.

Maybe I'm old but where exactly are the "dragons"?

How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?

If so, I'd like to see more design patterns around existing search problems:

- Correcting or backtracking based on feedback.

- Measuring relevance.

- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?

The article sounds like AI slop with some predictable tells like short punctual sentences, bizarre jargon, and titles like "Recipe 4: On-The-Fly Embedding (The Fresh Data Play)"

Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"

All computer primitives are relatively straightforward in pure form and vastly more complicated in real-world scenarios.

But over-engineering things is fun.

RAG is one of those things where I can hyper optimize to an absolutely needless degree.

I don’t understand the 4th option, “on the fly”. It didn’t seem to be explained properly.

Start with BM25 and only add embeddings when keyword search actually fails you. Saves a lot of pain.

Agreed, simpler is almost always better. The hard part is resisting the urge to over-engineer it.

Agreed. Embeddings are cheap to try and hard to mess up. Most projects can do plain semantic search first and see if they ever need more.

I skimmed through the article and it seemed okay. But then I lost my enticement when reading the comments saying this is an LLM written article.

The easiest way to strip complexity is to expose simple tools to an agent model like SID-1 that can use them well. It makes more of an effort for hard questions, and little effort for easy ones.

(found of sid.ai so obv biased)

Only those who mastered the craft makes their work look simple.

The AI that wrote this might be the master not the writer, as this looks written by AIs.

I will use the author's agents, not read his articles or use him for the job.

I believe embedding-based RAG, everybody is using, will end. As chips advance, you would use a big llm instead of word embedding for retrieval. It's much more accurate and extensive covering every topic.

Still need ~2 years to be replaced.

Very little of this is RAG but rather just FTS with clever reformulation and re-ranking.

RAG is about providing an grounded response, given the actual data in the corpus.

Great article and content, nonetheless!!

What RAG means for AI is what a library means for human beings.

It's necessary and would be good for you if you want to learn something systematically.

But for most of the normal issues, we can not rely a lot on it.

i want to ask that, if a user want to search sth, but he doesnt know the exact name(keywords), just some description. at this moment, whether the text serach fail?

Im curious whether the $10,000 figure includes unstated migration costs, since the raw embedding API cost under the earlier assumptions comes to $10.

Helpful framing, thanks for sharing!

Bookmarked, thanks again

OT but its interesting that none of the harnesses today use embeddings but just simple grep. I would not have predicted this

yet harder to implement proplery than you think

RAG is so 2024.

Wow! Terrible layout. Shouldn't fully justify on a small screen.

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[dead]

[flagged]

[flagged]

[dead]