No, the bias-making machinery is embedded in the machinery, part of the purpose of which is to do a rough kind of statistical analysis via "attention". If for example "Tufa" keeps appearing (n=small, but more than for the other fake tribes) next to terms indicating skill at some task, of course that will be noticed. It will last for as long as that information is in the context window (weights for the current model don't get updated as a result of conversation; that's just not how they work). And of course that can happen from random chance, and of course the LLM has no way to externally verify the extent to which randomness is in play (or the ground-truth probabilities).

The paper makes clear that they used pre-trained, frontier models — in other words, they did not train models on fake data about the fake tribes that would ascribe fake stereotypes to them. There is nothing to suggest that the training data somehow accidentally encoded biases related to fake tribes that the creators of the training data (i.e. ordinary people going about their ordinary Internet lives) somehow accidentally expressed.

There is also nothing to suggest that reading the entire Internet would somehow predispose the reader towards the general idea of being "biased", in the sense that you would have to have in mind to see an actual problem here. But really, the kind of "bias" we're talking about here is really pattern-matching on the available data, which is a big part of what leads people to apply the term "intelligence" to the models. See also the way that people try to make "culturally neutral" IQ tests specifically by having them focus on the ability to infer patterns (e.g. https://en.wikipedia.org/wiki/Raven's_Progressive_Matrices ).