> the concepts of tokens

Tokens are what you get when you have to map a chunk of text to specific numbers, so that the LLM can actually work with it.

https://platform.openai.com/tokenizer

Reducetively speaking:

[Your text] --> [List of numbers (tokens)] --> [List of [List of numbers describing a concept/term/thing/job/item/etc] (embedding)]

That final thing (List of embeddings) is what gets fed into the LLM, so that it can produce (List of embeddings most likely to follow afterwards), which is then turned back into text.

> and their worth

Tokens are priced based on the model you're using. In general, expect to pay more when you need to use larger models. (Note: Exceptions exist, but treat them as exceptions)

The space as it is now prices tokens based on:

- how many tokens it needs to process (input),

- how many it needs to generate (output), and

- how often you send the same chunks of text that their system already has on file (cached).

> and harnesses

A harness is something that adds in tools & directions for the LLM, so that it can do its tasks. In a dumbed-down semi-visual summary:

---

[Text about what the LLM should be & what they're allowed/not allowed to do]

[Text about available tools & how to call them]

[Your text here]

---

The harness, if its part of a program/app, then looks at what the models generate in response, and does the (file writing/reading, tool calls, web searches, subagents, etc.), based on the formatted output that the LLMs should produce.

> and what exactly all of those things accomplish

Depends on the goal of the user. Some use them as assistants, most use it for coding.

https://openrouter.ai/rankings?utm_source=share&utm_medium=c...