alt Hacker NewsYou will absolutely struggle to get all the info you need into 700 tokens per page.
Edit: There's also the added complexity of running a browser against 1M pages, or more.
You will absolutely struggle to get all the info you need into 700 tokens per page.
Edit: There's also the added complexity of running a browser against 1M pages, or more.
I agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go.
This library also supports HTML as input so running a browser is not required.