alt Hacker NewsAn LLM is a set of structured matrix multiplies and function applications. The only potentially non-deterministic step is selecting the next token from the final output and that can be done deterministically.
An LLM is a set of structured matrix multiplies and function applications. The only potentially non-deterministic step is selecting the next token from the final output and that can be done deterministically.
Matrix multiplication on GPUs is non-deterministic. As are things like cumsum()
https://docs.pytorch.org/docs/2.11/generated/torch.use_deter...
This comes down to map reduce and floating point's lack of associativity. You see the same thing with OpenMP on CPUs.
People are constantly claiming determinism in LLMs that is just not there.