alt Hacker News> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle.
Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of rules for letters and letter combinations which may or may not result in the same sounds, hand-coded as a huge if tree by a linguist not a programmer)? What about their relatives, do they match by the same rules? Should we incorporate domain knowledge about local naming customs? Etc etc.
I pointed a coding agent to this problem, and it aggressively started coming up with complex scoring rules and testing them against real datasets. Which led to sort-of acceptable results, but it still missed lots of cases which were obvious to a human, and had false positives which were obvious to a human. Which I could trade off, and slightly improve, with more back and forth with the coding agent.
Pointing a good LLM to all the information about two people, would of course give great results. Maybe even better than human judgment. But I can't do that for 100000^2 people, it would be too expensive in all sorts of ways. I need a fast, reliable scorer. I could maybe train an embedding, but that would be a huge job and where would I get the quality data?
Replies
You need blocking!
Fundamentally this is an entity resolution problem. An LLM can score pairwise really well but scoring all the pairs would be insanely computationally difficult.
If you can constrain the set of potential matches up front by querying the dataset for things that could be matches it gets a lot more tractable to use an LLM for this.
Are there any heuristics you can use to reduce the search space? You mentioned soundex transformation and maybe prefixes of last names could work? Even if you get the number of potential matches down by a few orders of magnitude this gets more reasonable!
Check out https://moj-analytical-services.github.io/splink/index.html
Maybe you need Prolog instead.
There's also two other important limitations to using an LLM and just providing it with pairs of records.
It does not know enough about the records in the context of the overall dataset:
- what is the data quality and to what extent do we expect a errors in some fields - how unusual are certain values such as names in the context of the dataset as a whole, e.g. some names would be very common in some countries but rare in others.
I've written in more detail about this here: https://www.robinlinacre.com/fellegi_sunter_accuracy/