alt Hacker NewsThis is a good example of the difference between an LLM providing a theorem and a person deducing theorem, and all the juicy knowledge we get from truly understanding things: the Google SWE found something he thought was undiscovered. He asked AI to make an image what it might've looked like and speculated that it was used by the local elite.
But Dr. Estrada-Belli found something that was undiscovered. Finding this structure wasn't his number one goal - it was the side effect of a larger anthropological assessment that allowed him to improve estimates about population density of the north versus south, how Mayan homes and farms are distributed, what this means about social relationships between the elite and common people, how different regions were interconnected, etc.
Interestingly, one of Dr. Estrada-Belli's primary conclusions is not that these structures were only used by "the local elite," but that the closeness of residences and farms to these community sites suggests that the elite and the common people actually had a much closer and nuanced relationship than we previously thought. That's the opposite conclusion you'd get from the Google SWE's assessment.
Knowing that a Mayan site exists is one thing. Understanding what it means is another.
Replies
Very reminiscent of my experience pointing persistent agents at computational physics problems. It can find amazing results and really surprise me, make a write up, and convince me it's a real discovery... until i tell it to do a literature search, and be meticulous. Then it comes back saying "prior art has reduced the novelty of the discovery", and finds papers explaining the discoveries very easily and intelligently.
Perhaps google SWE should have requested a literature search before publishing his results as novel
This is also a good example of the disconnect between what's valuable to scientists, and what members of the public think is valuable.
In this blog post, we have LLMs finding just one site in a 1,220 km^2 search area. And then sent an email to a government agency and got a polite "cool story bro" email in response. And that's probably as far as this will go because, in mesoamerica, "undocumented" sites are a dime a dozen.
Compare, for example, with this paper in which researchers found 6,764 sites in a search area only 1/10 as large: https://www.cambridge.org/core/journals/antiquity/article/ru... , and then proceeded to do real science with all that data.
And I have to point out, part of why they were about 5 orders of magnitude as effective at finding these sites is that they used much more sophisticated methods than what the LLM was able to come up with.
IMO, his primary mistake was not turning the agent swarm on a proper research task that exhaustively determined whether it had been discovered or not.