logoalt Hacker News

Xyrayesterday at 9:29 PM1 replyview on HN

making the internet sql queryable, crawling, cleaning, indexing, embedding many sources into source-aware schemas, solving contention problem with free-floating pricing.

Currently crawling over 1M records/sec. software is still in alpha.

scry.io.

goal is a 10PB NVMe cluster online by November (need funding champions) as a public benefit project, so prosocial researchers and builders and their agents can have low-friction access to running analytical queries over the public internet.


Replies

chickensongyesterday at 11:21 PM

Do your crawlers respect robots.txt and have a fixed IP pool that can be blocked?