Chomsky and the Two Cultures of Statistical Learning (2011)

Norvig (and Chomsky) are talking at cross purposes.

Norvig confuses engineering for science. If an engineer wants something that works (and guarantees that it will work) then you use the most effective tool available to achieve that goal. No problem there. But there's absolutely no reason to think that tells you anything scientific about the phenomenon you're modeling. For example, aeronautical engineering says very little about how birds fly. And - currently relevant - LLMs learn to use and understand language, but they use millions of times more training data, and millions of times more computing power, than a child does. So an LLM may be a great technology, but it's probably not a good model for how humans learn and use language.

Chomsky OTOH doesn't see any use for statistics in science, at least not in linguistics. Fortunately for the field of linguistics, Chomsky's influence in linguistics is waning and linguists are beginning to use statistical tools to do science.

> But the vast majority of people who study interpretation tasks (such as speech recognition) quickly see that interpretation is an inherently probabilistic problem: given a stream of noisy input to my ears, what did the speaker most likely mean? Einstein said to make everything as simple as possible, but no simpler. Many phenomena in science are stochastic, and the simplest model of them is a probabilistic model

Why the "simplest" model? Is a probabilistic model not the only model able to deal with uncertainty? What other kind of models are possible to represent uncertainty and inherent ambiguity that are not probabilistic?

It's more or less the classic clash of two modernist attitudes, that of the rationalist (Chomsky) and the empiricist (Norvig). The intellectual wound this division introduces is longstanding. However, I would say Norvig has the overall advantage.

1. Norvig is correct that observation and fact/data collection are necessary for science. However, Chomsky is correct that the ultimate goal of science is to determine the principles governing something. Ergo: you need the first to do the second. The intellect abstracts from source data, but nothing is in the intellect that is not first in the senses.

2. Chomsky here is uninterested in things like speech - he only cares about the underlying syntax. But syntax - expressed and instantiated in matter - is known through speech, and probabilities presuppose an underlying determinate syntax. Ergo: a complete science of speech is end-to-end, from the "reading" or interpretation of noisy data through syntactic processing all the way through to expression.

3. Chomsky focuses heavily on internal generation. Norvig points out that interpretation is a separate, highly practical challenge. Figuring out what someone means in a noisy, ambiguous world is an exercise in prudence which involves weighing contingencies and probabilities to arrive at the most likely truth. Of course, this can only be done against a presupposed acceptance of the function of speech which is to communicate meaning and what is true (hence why lies work and why they are an abuse of speech). Probabilistic reasoning is still reasoning. It's reasoning in the face of incomplete information (against a backdrop of presuppositions) which is a component of prudential reasoning.

This is an old, old post by Norvig fyi.

Really great post, though a bit old, still relevant. Peter Norvig is one of the better thinkers of our generation.

Have to say that I confronted myself with the fact that I can't read longer posts anymore due to overuse of AI. Guess a detox is required...

It's so clear that Norvig knows so little about linguistics as a subject, I mean the suggestion that English is "kinda" a pro-drop language like Spanish is just unserious. There's this kind of lazy criticism of generative work where people hear the vague idea of a theory and critique without even seriously reading anything about it. These are facts that generativists are perfectly aware of and have explanations for.

It's sort of like criticising Galileo's putative experiment about dropping two balls from the leaning tower of Pisa by pointing out that a feather indeed falls slower than a iron ball.

[dead]

[dead]

[dead]

[dead]

chomsky was also a close friend of epstein and raped kids