alt Hacker NewsWow, will they offers some sort of reduce a web page into a markdown file api as welL so we can get full web pages at reduced token sizes?
Replies
moealmaw • today at 5:18 PM
They already have that[0] it’s called Markdown for Agents and it works on any URL
[0] https://developers.cloudflare.com/fundamentals/reference/mar...
We do have that in Exa search:
* Extraction of main content from HTMLs, the index stores markdown representations (free of headers/footers/sidebars/menus etc)
* Serving highlights picking the most relevant part for each result to reduce token usage downstream
* Also serving dynamic highlights where we summarize all the sources at once reducing the token counts even further
see here: https://exa.ai/docs/search/highlights and https://exa.ai/docs/contents/quickstart