Bag of Decisions Reranker

> 1 - Is the document about desks designed for children?

> 2 - Does the document offer products or information on kids’ desks?

> ...

> 9 - Are brands or models of children’s desks listed?

> 10 - Does it cover where to buy or how to choose a desk for kids?

>

> Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance

It seems to me that this is a rediscovery of natural language inference (NLI), which was very active in NLP research pre-LLMs. I have a side hobby of training NLI cross encoders, and I'd bet that many of these use cases could be serviced by a good quality small NLI model.

See: https://huggingface.co/dleemiller/finecat-nli-v2-l

And also: "Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI" https://www.semanticscholar.org/paper/Less-Annotating%2C-Mor...

He compared techniques that used:

- BM25 only

- Jev only

- LLM + Jev

The latter worked best.

But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.

EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.

A possible improvement could be generating a sensitivity score for each question. Some are more important than others. For example not mentioning kids at all should cause a huge decrease in relevance, but not mentioning a specific feature should only cause a small decline in score.

This is a cool idea. It would be interesting to see how sensitive the results are to the strength of the LLM used for query augmentation. They used GPT-5 in their experiments, and generating all those tokens would presumably add a lot of latency in your query pipeline.

Another baseline that would be interesting to see is just pure LLM reranking--is Jev/the query augmentation adding anything? Presumably the idea is that Jev is cheaper, but even with a costly LLM you might get a lot of cache hits as long as the query (the variable portion) goes last. Even setting caching aside, almost the entire request is prefill plus a very small amount of generation for the judgment itself. You may end up generating fewer tokens just using the LLM as a judge directly than you need to generate the rubric for Jev. OTOH if queries can be normalized or are naturally repetitive you may be able to cache the rubric...

Have used decision models the same way: Had LLM generate/tune questions for Clef to label/extract emails with {useful work thing} content

Too early to say how useful this is versus existing techniques, the cool thing is this is the first thing I've found local models genuinely useful for. I think I was able to process 50,000 email questions per hour or so with Clef flash on Strix Halo with a custom engine. A modern gaming GPU would likely be much faster

Shouldn't you multiply instead of summing the scores?

Would you show a result with 3 really high scores and one which is really low?

Super interesting. It’s amazing all the applications we are starting to see. Even if not optimal, a lot of people are now thinking about options and workflows that they weren’t before