alt Hacker NewsLLMs are non-deterministic, you could run the same prompt on the same code base with the same models, and have different result. I don't think only one run with each model is enough to draw conclusions.
Replies
Eggpants • yesterday at 6:05 PM
Honest question about the non-deterministic nature of llms. Every programming language provides some way to control the random seed. Is this not true in GPU programming land?
➕ show 2 replies
We have to keep remembering this for some reason.
New model comes out and someone says, "it can (or can't) draw ducks on bikes" or some esoteric thing. But it's not enough statistical power to claim anything other than a personal anecdote.
I got a flat tire in downtown today. Therefore, driving in downtown causes flat tires.