Illusions of understanding in the sciences

Looking at the paper, the core message is 'that even scientists harbor the illusion of understanding more than they actually do'.

In reality, science operates much like a mental model. The paper argues that just because a model predicts future values more accurately, it doesn't mean the model explains the actual causal structure. Yet, the fact that outcomes fall within the predicted range reinforces the illusion that one has truly 'understood' it.

This reminds me of the statistician's aphorism: 'All models are wrong, but some are useful.' Science itself, in a way, is a mental model—a simplification created for humans because the world is a complex system that is cognitively impossible to fully comprehend. Within that framework, certain facts reinforce the mental model, while others weaken it. While mental models vary from person to person, in a broad sense, we are commonly taught to view the macroscopic world through the Newtonian model and the microscopic world through the quantum mechanics model.

Reading this makes me reconsider what 'understanding' truly means. I believe the starting point of genuine understanding is acknowledging that perfect prediction is ultimately impossible, and that when viewing the world through our mental models, what matters is defining what we consider to be acceptable 'lossy information' (or information we can afford to lose)

The authors of this paper have not studied what historians and philosophers of science have written. They just use 'induction', 'validity', etc. They reinvent the wheel. They write "Of course the validity of that induction depends on a host of other assumptions.". Duhem-Quine thesis is better than this way of formulation, as the latter doesn't use 'validity'.

If authors ever come to this forum, please read Duhem-Quine thesis, over/under determination, inference to the best explanation, Goodman's paradox, also how various theories in philosophy of sciences: from Popper to Kuhn, Lakatos, Laudan, etc.

This is kind of interesting, but I predict that it pleases almost nobody. Philosophy of science types will be kind of annoyed at the preoccupation with statistics, ML people will be annoyed at too much philosophy of science, etc.

I totally support a goal to get those groups talking more but something tighter is probably better. And why isn't it tighter? Without big original contributions, the goal does seem to be a survey

This is a classic case of overthinking. Induction should not yield new knowledge because nothing new is discovered, but it does. Deduction likewise also cannot establish new knowledge, yet it does. Empirical science is flawed on extremely many levels but it works because on average, over time, many converging observations can build refined and accurate causal theories. It’s a matter of practicality that things cannot be proven fully. Judging from the state of modern medicine, engineering and the sciences, the system works ok regardless

It's funny when you think you know something pretty much thoroughly. Then you learn a bit more and realize that your understanding was a bit simplified, had gaps or there was a whole other level to it.

The feeling is a strange mixture of disappointment, awe, annoyance and excitement.

More predictive power is always a good goal, full stop. This is orthogonal to whether the model producing prediction helps with "understanding" directly. Predictability encodes understanding in a strict information theoretic sense, regardless of our ability as humans to access that understanding.

> Many people know of Simpson’s through simple examples. One was UC Berkeley admissions showing that individual departments admitted more women than men but the university as a whole had admitted more men than women (Bickel et al., 1975).

This only seems possible if students can be admitted to more than one department.

got me at "Most often scientists believe they understand more than they do, making their belief an illusion." but why is it still bothering me? 1. feels unfalsifiable in spirit 2. somewhat restates "all models are wrong, but some are useful" less cleanly 3. doesn't really offer like, what can we do as science people? tomorrow morning perspective

Is it a probability that the authors understood the notion of Understanding all wrong?

;).

What is a model anyways? There are so many answers to say you that. The models are almost the same models, but at a different abstraction away from the original experienced in reality.

this is extremely long and repetitive.

"the sciences" is very broad. in biology there are established methods for establishing causality (i.e. Koch's postulates, etc), and even then conclusions are generally qualified. not sure about the other fields, but I wish they had more concrete and recent examples of what they are talking about. this was painful to even skim.

also for some reason i cant click on anyting on the site or select text?

This applies way beyond the sciences. So many people now think they understand something because they can prompt an AI to give them the right answer. Thats literally this same illusion just with a new interface on top. Getting correct outputs isnt understanding.

So, to summarise, consistency is the virtue of a narrow mind?

Popper writes the philosophy of science in a Platonic micro-descriptor fetch, which is 20:20 recursion.

Like thinking LLMs aren’t magic* because you utter “it’s just predicting the next token!” I’d argue, only slightly tongue in cheek, that thinking of LLMs as magical leads to more effective use than the predicting-next-token explanation.

See also Frank Keil’s “illusion of explanatory depth.”

* magic not as “unreal,” but in the classical conception of a living magic world where mental intentions can manifest physical realities

> It is the writer's experience that new degrees of comprehension are always and only consequent to ever-renewed review of the spontaneously rearranged inventory of significant factors. This awareness of the processes leading to new degrees of comprehension spontaneously motivates the writer to describe over and over again what—to the careless listener or reader—might seem to be tiresome repetition, but to the successful explorer is known to be essential mustering of operational strategies from which alone new thrusts of comprehension can be successfully accomplished.

R. Buckminster Fuller – Synergetics: Explorations in the Geometry of Thinking

> Delusional interpretation is a false deduction drawn from an accurate perception. The subject perceives correctly, but reasons wrongly; in him, judgment is impaired by affective disturbance, while the senses remain normal.

> Delusion progresses by accumulation, radiation, and extension; its richness is inexhaustible. The plan of the edifice does not change, but its proportions keep increasing.

> Every new fact, however insignificant, is immediately incorporated into the delusional system, where it becomes a fresh piece of evidence. The patient lives in a state of perpetual suspicion, searching everywhere for guiding threads, clues, correlations.

> Interpreters are not hallucinated subjects; they are logicians gone astray. Their point of departure is an intuition or a false belief, but the consequences they draw from it follow one another with an apparent rigor that often deceives the superficial observer. It is order within madness, logic in the service of the absurd.

> The need to write, graphomania, is in many interpreters a major symptom. They accumulate immense files, endless memoirs, interminable correspondences, in which every detail of their existence is dissected, analyzed, turned over and over, in order to bring to light what they believe to be the truth.

Sérieux & Capgras — Reasoning Madness: The Delusion of Interpretation

> The madman is, rather, the free man: the one who does not allow himself to be chained by the false appearances of common reality. Delusion is not an insult to logic; it is logic driven to exasperation. The paranoiac is a tireless translator, a man who spends his life deciphering the signs of the world in order to find in them the key to his own destiny. Far from being chaos, psychosis is an attempt at rigor, a complete theory that the subject constructs in order to account for his own genesis and his place before the Other. The risk of madness is measured by the very attraction of the identifications through which man alienates his freedom.

> following Fontenelle, I surrendered myself to that fantasy of holding my hand full of truths, the better to close it over them. I confess the ridiculousness of it, because it marks the limits of a being at the very moment when he is about to bear witness. Must one denounce here some failure in what the movement of the world demands of us, if speech was offered to me once again, at the very moment when it became clear even to the least perceptive that, once again, the infatuation of power had only served the cunning of Reason? I leave it to you to judge how my inquiry may suffer from it.

Lacan — Remarks on Psychic Causality