making the internet sql queryable, crawling, cleaning, indexing, embedding many sources into source-aware schemas, solving contention problem with free-floating pricing.
Currently crawling over 1M records/sec. software is still in alpha.
scry.io.
goal is a 10PB NVMe cluster online by November (need funding champions) as a public benefit project, so prosocial researchers and builders and their agents can have low-friction access to running analytical queries over the public internet.
Do your crawlers respect robots.txt and have a fixed IP pool that can be blocked?