LLMs dont do gradient descent to generate tokens.

They are trained by gradient descent, but inference doesnt involve it.