logoalt Hacker News

slowinyesterday at 9:57 PM3 repliesview on HN

Presumably markdown uses far fewer tokens.


Replies

o_mtoday at 5:17 AM

Markdown isn't as expressive. Not all HTML content can be converted to Markdown without losing some of the semantics

honryesterday at 11:07 PM

Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.

show 2 replies
simonwyesterday at 10:12 PM

That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.

show 2 replies