I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".

It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.

It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.

I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users.

(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)

I indexed thousands of documents into a SQLite database with an FTS5 index, plugged it into DeepSeek v4 Flash and got better much better results than any other commercial solutions my company has tried in the past.

The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.

This depends on your users and doc corpus. On finance docs and with users that use jargon and acronyms heavily, BM25 can fail on trivial queries... you end up encoding a masters in economics into the query re-writing logic.

Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)

RAG is art. I have a very straight forward setup that is highly modular.

RAG is routing and decision making.

I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.

I agree with you.

Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.

It maintains itself.

Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.

I found my niche with RAG selling and I build them myself.

I take pride in them.

So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.

And there are regulated businesses where certain information is required to be always factual correct - pricing for example.

Vector search becomes a liability for this use case.

So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.

I love this kind of stuff.

And there is personal information etc.

Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.

And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.

It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.

I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.

RAG gives me dbCORE vibes so to say.

Good luck and fun with your RAG systems.

Re: the rube goldberg machine of diminishing returns

https://www.anthropic.com/engineering/contextual-retrieval

This is from two years ago, but I think it's still SotA?

I thought text search was always the first thing you try, then fuzzy search, then you go for RAG

Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?

And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.

How long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.

> people vastly underestimate full text search

It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.

RAG only makes sense if you have an LLM review the results, pick the most relevant ones and iterate further if there's a need running another query and repeating the process. Raw dump of vector search (even with reranking) is asking for troubles (or rather weird user questions like 'why this crap popped up in the results?')

On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag

I believe the second Suggestion solves 95% of my problems. I want a system where i can describe my search and the system generated 5-15 keywords for a query

Yes, and don’t forget, LLMs are very good at tagging, so it’s not even that painful to backfill the corpus.

I don't know what level of quality is required for this site but RAG is trash its just trash. its magic beans.