My understanding is that to unambiguously quantify information, you need to have a known model within which that information fits. In this context, talking about LLMs adding value (or not) via inserting new semantic information, we're assuming that public knowledge on the internet is not new semantic information, and the quantity of information of contained in text talking about public knowledge is equal to the ~32 bits needed to point to that knowledge.