alt Hacker NewsI’ll leave it to other comments to discuss the societal and moral implications of being able to do this (which, I agree, ick…). On a practical level:
We train an encoding model, a “digital twin”, that predicts how each visual region responds to any video. Now we can ask: which video would make a chosen region light up the most? NEvo searches for that video automatically, using the twin’s prediction as its reward.
I only scanned the paper, so maybe I missed it, but is there any confirmation that this ‘digital twin’ works? Like, do the generated videos actually cause the same patterns as in the ‘digital twin’ brain model in real humans in an MRI machine? My instinct is to be skeptical that it’s possible to reliably create a video -> brain activation prediction model.
Replies
... and shouldn't this backfire hard?
All the signals that are missed by the (f)MRI are never "mirrored" in the digital twin even though on screen it will look like it. Bam, Experimenter Bias in the machine and I don't know how to phrase it but ... does this method/experiment leave any wiggle room for Falsifiability?
Even if the Hawthorne Effect does not apply to humans, it most certainly translates onto the brain and sensory (post)-processing-- live and remembered (vs in-memory, stored), real, virtual or imagined.
... I just realized how vastly different peripherals are when real, remembered, virtual or imagined sensory input is (post-)processed ...
It feels more like the machine will reproduce a superficial pattern. And at least some signal streams echoing in the brain on input from the digital twin won't emerge because the digital twin has only part of the data. And that's a premise for an unnoticeable rewiring of what once used to fire together, ... given a hell of a lot of exposure, of course, ... or not ... let's see
EDIT n: the paper says pretty much all that in like ... science, bitch!
Epic stuff.
A. Unless you think that brain science is immoral science, then I do not see any problem with this kind of research. As a neuroscientist, I strongly object to the insinuation as unfounded.
B. "Digital twins" e.g. [1] are a growing class of brain simulations that can successfully approximate brain activity patterns at large scale. I think these can be very useful, but we shouldn't think that they are at the level of actually simulating a brain. They are usually made of model neuronal approximative simulations (e.g. integrate and fire, balancing excitatory and inhibitatory neural populations within units), then using diffusion imaging to estimate white matter axonal wiring between those populations from the subject to increase the accuracy of the simulation. These are increasingly being used to, for example, model how a surgical intervention would effect seizure propagation prior to actual surgery. Here is a nice episode of Theoretical Neuroscience podcast [2] on the Virtual Brain [3], one of the available models for this kind of work.
C. In terms of validation. Only partly. From my quick read, this NEVO model optimized neural response only in the digital twin encoding model. While the digital twin model reportedly has solid predictive validity [4], which by the way was not the Virtual Brain model I mentioned in point B. Moreover, the outputs looked neurobiologically plausible, but at this point, there is no independent model or new fMRI showing the optimized stimuli actually drive the target regions. This was performed using previously collected fMRI data, and full validation of this model *IS* the obvious next step, but the money to collect such data does not come from nowhere: funding will be needed, and such a paper as this can help them get it.
D. A final point I'd make. We have long been able to create static stimuli that we can be fairly certain will activate above baseline certain brain regions, on average. Certain stimuli-region pairs ar emore homogenous between people, others e.g. the fusiform face area (FFA), are small enough that individual differences prevent a simple ROI approach, and identification depends on using face stimuli to identify at the individual level, but for the most part, it is reliably locatable. Brain activations are very coarse things. In fMRI, you are talking about ~3x3x3mm voxels (27mm^3) where the hemodynamic responses have a ton of spatial autocorrelation, or in EEG, where the surface spatial area of the reeptive fields are very large(~400 mm^2). These virtual twin models already do a decent job of modeling dynamics of the brain there parameters are tuned to *at this scale*..but this scale does not have a ton of information content. Automating this with video content is not that much a reach.
[1] https://spj.science.org/doi/10.34133/icomputing.0055 [2] https://theoreticalneuroscience.no/thn23/ [3] https://www.thevirtualbrain.org/tvb/zwei/ [4] https://www.biorxiv.org/content/10.1101/2025.07.22.664908v2....
You would like to buy some paperclips.
Yeah, this smells very fishy to me. It's almost trivially easy to gather a small validation dataset in humans for the paper. At my institution, it's about $800 an hour to scan someone. You can probably get enough data to validate the model with a half hour scan. Surely the group has enough grant funding to pop a few healthy controls in the scanner.
I haven't looked in super close detail to the paper, but their methods section says that they fit a video model (V-JEPA2) to the fMRI dataset in a voxelwise ridge regression, meaning that the baked in assumption is that the visual response affects each voxel independently. Voxelwise models are very nice for making statistical inferences, but are less good for prediction and modelling tasks, because our brains certainly do not work as collections of independent regions.
BOLD is intensely messy data, and their design is far too simple IMO to reflect anything of reality.