I think you are just litigating the philosophical debate of Bayesian vs frequentist interpretations of probability. Because the weights really are just probabilities.

Bayesians say that the probabilities represent strength of belief, implying some subjective knowledge or information. It is necessarily subjective in that it requires priors, i.e information the predictor knew before making the prediction. In other words, the LLM has priors from training and is predicting tokens using real knowledge

Frequentists would say that probabilities are simply objective facts - e.g we all agree that the physical property of temperature follows from any molecules matching a particular energy distribution. You’re not predicting anything, there’s just some outcomes that are happening at the expected rate. In other words, the LLM is a stochastic parrot/next token predictor