alt Hacker NewsAll these weird partial language thought patterns im surprised none of the teams have taught the models to think in something like court stenography or some very dense pattern (i mean they even tried caveman language at one point)
Replies
robkop • today at 8:42 AM
One of the main goals at the moment is to keep thinking human legible. You can imagine how much harder it’d be to do root cause analysis on the recent OpenAI event if we couldn’t even tell what they’re thinking.
Keep in mind that the model is thinking in a token space, itself a compressive representation of language.
(Note: there's still a huge grammar penalty, so, ugh do think small.)