logoalt Hacker News

smallerize • yesterday at 9:17 PM • 2 replies • view on HN

I'm being very literal about the randomness. The language model outputs a distribution over all possible output tokens, and the chosen output is sampled from that distribution. The process intentionally includes output that is not the top-ranked token according to the model.

Replies

perching_aix • yesterday at 10:34 PM

Think about that for like two seconds.

Yes, randomness is involved, except you're destroying the snot out of it by biasing it to hell and back with those billion-trillion parameter models in front of the sampler.

Alternatively, check the difference between a typical ChatGPT prompt response, and the output of the following:

  import tiktoken, secrets
  
  enc = tiktoken.get_encoding("o200k_base")
  tokens = tuple(enc._mergeable_ranks.values())
  msg_len = 128 + secrets.randbelow(2048 - 128)
  token_ids = [secrets.choice(tokens) for _ in range(msg_len)]
  print(enc.decode(token_ids))
You'll note that they're not much alike. You may also note that with LLMs you can set the "temperature" parameter to zero, in which case the sampling is no longer randomized. So much so, that given various runtime conditions, passing in the byte exact input will return the byte exact output, every time. Possibly across machines, depending on the inference engine used.

Even if you mean it on a "higher level", imbued into the language emitted, do feel free to ask a model with disabled tool calling to give you some random numbers / words. You'll find that it isn't very random. Certainly wasn't when I last tried this for entertainment.

➕ show 1 reply
antonvs • yesterday at 9:33 PM

No, you’re not being even slightly literal. If language models were random number generators, their output would be gibberish. That’s literal.

➕ show 1 reply