logoalt Hacker News

gavinray • today at 4:29 PM • 2 replies • view on HN

I want to attempt an explanation as it seems a lot of folks here didn't grok the very terribly written README

"Auto Ontology" is sort of a poor name.

Here's the problem:

Your company has data scattered across multiple types of databases. People who make decisions want to ask questions about that data.

The old way is they would go to a Data Analyst, and say, "Get me a report on so and so", and it was the Analyst's job to iron out the right series of queries and one-off scripts to generate the needed info.

With a system like this, the idea is that all your database models have been introspected and stored in a central metadata catalogue.

When a question is asked like "What were the top selling items in the month of May?", the metadata catalogue tells you "Products are stored in FOO_DB" and "Product_Sales are stored in BAR DB" and so you go and fetch the right data from the right place and do whatever in-memory calculations you need to do.

What the system provides is the implementation code for:

- Introspecting each of your DB's

- Aggregating model + metadata information in a central, virtual DB (VDB)

- Generating dialect-specific SQL from natural language queries and routing them to the right DB for the underlying table

- Combining results in-memory when a natural language query requires multiple datasources

- Handling proper permissions, etc

A GOOD system for doing this also integrates with your external API sources (OpenAPI, GraphQL, gRPC, whatever) and also documentation sources (cough, I work on this, cough).


Replies

ewild • today at 6:06 PM

what happens when your column names dont accurately represent what data is being stored (i know i wish it did but it doesnt at my place right now) and its all esoteric knowledge held by our data scientists? my worry is now the guys above will think its answering their question without knowing that its either A. wrong or B. data they may not be able to use for legal reasons on the project they are trying to make.

➕ show 1 reply
throwaway5752 • today at 4:32 PM

"seems a lot of folks here didn't grok the very terribly written README"

I tried to make it through the README and it was just unmotivated slop. Perhaps it reads better to agents. It is like the Van Halen rider story - if the authors didn't care to write a good README, what are the odds that they paid attention to the rest of the large codebase?

➕ show 2 replies