alt Hacker NewsA misalignment of AI in mathematics
Comments
I find this explanation has a lot of applications for programmers within companies. It’s one thing to get your LLM to give an answer, it’s another to bring a group of people into shared understanding of a domain.
Is this really any different than the problem in software engineering - where AI is doing the work of junior programmers and now they aren't getting the development they need?
Seems the same to me. And it'll be the same in all industries soon enough. And then it won't just be the junior people.
All the same problem: what do people do now?
I note here an isomorphism with my essay from earlier this year, The deliverable is you! Programming as theory building:
A program is not the only output of programming. The other, arguably far more important output, is the programmer.
When you write the program — with your own hands — the program is proof that you have a solid mental model of the program.
When you let the computer write the program for you, the program is proof of … nothing.
https://nekolucifer.substack.com/p/the-deliverable-is-you-pr...
If you internalise that AI might actually reach super intelligence then logically the question becomes "so what exactly are humans for if literally everything can be done better by machines?". Then mathematics and all intellectual work, as argued for here, becomes quite clearly a recreational pursuit.
A Severe Misalignment of AI in human-centered Mathematics
Somewhat unrelated: is it wrong to say mathematics is not art, and that there is always a right answer? I know that's not romantic, but maybe it's true.
Before LLMS, programming was something I might've said required creativity and human input to do properly. It's not that creativity or human input isn't valuable anymore, but AI has forced me to realize that coding is much a means to an end, and that all things considered, the end matters much more than the means.
If we can make important mathematics progress faster and better with LLMs, I think it's wise not to fret over an apparent loss of our humanity. Perhaps that's only a loss we want to have.
It's good to start a conversation, and the number of Fields Medalists behind this certainly lends a lot of weight to it. But I'm not seeing a strong argument for misaligned incentives beyond the specific plagiarism allegations. The job of an AI company is to build systems that solve problems. The job of a mathematician is to advance the state of human knowledge. If anything, an influx of solved problems should increase the demand for human mathematicians who can convert them into conceptual understanding.
There's a reason mathematics is generally within liberal arts programs rather than science programs. Mathematics is the art of logic. Yes, sometimes mathematics becomes incredibly useful but most mathematics is never applied.
Compare that with computer science. Most of the work we do in software engineering is in service of an applicable output - software products that facilitate processes or bring in revenue. Turning up the dial on AI gets companies to these outputs faster.
Turning up AI on mathematics helps solve conjectures and can provide new insights. But it has a major misalignment with the purpose of mathematics which is largely intellectualism.
I have never, and I mean never, seen such a declaration have any effect whatsoever.
Contrarian take: I think the ability of AI to produce valid mathematical proofs (even inscrutable ones) is absolutely fantastic. Mathematics as a profession does not have a monopoly over math itself any more than professional pianists have a monopoly on who plays piano, when they play, and how.
I have sympathy for any jobs that might be affected (much as my own job has become more tenuous in software engineering). And if the field is disrupted by chaos that makes the research process unproductive, that's bad too and should of course be handled by applying better organization within the institutions that tend to perform mathematical research.
But to a large degree, the notion that "sloppy AI proofs are bad for mathematics research" seems like a total failure of the imagination to me. Attempting to find shorter proofs or more elegant proofs can be turned back in on itself via proof theory. There are proofs in Presburger arithmetic that are doubly exponential in the length of the sentence. Yet a more powerful theory like PA makes quick work of such theorems. The explainability or "subjective beauty" of a proof can be quantified and optimized against. Optimization itself can be optimized against. I really don't understand how this magical ability to know the truth of more theorems much more quickly—even via an "ugly" route—is anything but a net positive.
Seeing how /r/singularity and /r/accelerate are leaking into maths forums, I foresee a wave of comments that fail to understand Tao's message, whether on purpose or not, so let's try to be clear here:
Tao is not someone who is anti-AI for the sake of being anti-AI. He has been advocating for the usefulness of AI in maths for a long time, to the point that people have started calling him a shill for the commercial companies.
And everyone agrees that there are plenty of use cases to be had; helping with less interesting tasks like easing literature review, efficiently delving into existing work, doing review, whether on your own work or that of others, prototyping algorithms in areas where computation is useful, but also more in hands-on aspects of maths like validating potential proof directions by getting quick feedback on veracity of lemmas, etc., and, on very rare occasions, being able to one-shot the problem you care about.
The point he is trying to make here is much more subtle than "AI bad", and it's probably easy to miss if you have never engaged with research in maths: it's that the particular approach that large commercial companies have opted to take to produce marketing material can be a net negative. There is not doubt that -- even if you ignore the rampant plagiarism that has been reported across multiple problems now, the unethical attempts to oust authors, the outrageous attempts to scoop researchers instead of collaborating with them and building on existing projects -- it's nifty to have a machine that can help you figure out if a proposition is true or not. But just figuring out as much was never the point. When people have built problem lists, it's because some problems are more likely than others to provide new insight, and that insight is the target. And to than end, a poorly written paper with inadequate references and a pile of Lean is not valuable at all. Yes, now we know with higher certainty that Fermat's Last Theorem is true, but everyone expected that already.
One place where "just" answering the question can be a net negative is because the current incentive structure is set up in such a way that going in afterwards, trying to reclaim and extract the insights from a brute force solution, is considered less valuable work than that of coming up with a solution in the first place. That's a problem of incentives, and something Tao himself has addressed in e.g. his ICM talk, and that's something that we'll want to do something about. Until a better structure appears, though, if any given commercial provider of large language models really wants to help out with maths research and not just make more pre-IPO marketing material by competing with their customers, they could do so by using their magic machines to help build insight instead.
Maths will continue as a field of natural science in understanding the results and uncovering meanings in them. It is normal that the established community is afraid of the change, because it’s their _home_ that’s changing. But it will be a better home to the new generation nonetheless, one that’s not as daunting as the higher maths has always been to many. The concerns raised here will not be a problem at all.
Everybody is okay with coders losing out to AI, but when it starts hitting their field then there is outrage. I love it.
So what's the actual point here? It's too fast and we can't keep up?
Isn't that just a function of the technology itself, and the same problem being faced by every other field? And going to get exponentially "worse" every year!
Or is the issue that they're bad at explaining things, in a way that produces actual learning? (e.g. AI is amazing for learning but the net effect on students so far appears to be negative.)
I kind of expected a sober stoicism from mathematicians. Feels silly in retrospect. This is just the math version of the "anti-ai" movement by "artists".
The AI driven mode collapse of human thought advances. I am no skeptic or anti-AI, but this is definitely a concern I share. You even notice it in normal mundane tasks like programming, never mind the AI generated prose that we at least have become somewhat allergic to.
It wouldn't be so bad if you could just sit it out and say "Oh well, once the labs get bored with marketable domain X, humans will remigrate and re-apply creativity to it", but by then the damage might have been done and a field destroyed as an occupation. I don't know what to do about it, but I appreciate calling out the cynical tone-deafness of the AI companies here.
Jacob Tsimerman fields medalist notably missing from the list now works at OpenAI but recently admitted to be "grieving" for mathematics
On the contrary to what Tao believe, it seems like we need AI to move the needle on mathematics.
> problems in many fields of mathematics
Developing these different fields moves complexity from the field itself to the interactions of these fields.
Getting too preoccupied with the established terminology risks us a local minima.
Anf because the field overall has become so complex that we need to decompose into subfields, there will be a good chance that we will not, as individuals, have the capacity to truly see progress.
The map has become so big that we need better tools to work with it.
While I agree with this and appreciate Tao and other mathematicians to take the time to do this. There are similar concerns for many many other fields aka there is a general misalignment of technology. Take Software engineering for example, I can't believe there is a class of software engineers who wake up everyday and tell themselves, "today is the day I am going to automate the rest of my job".
Lets forget the hyper intellectual fields like maths and software engineering for a moment. What about taxi drivers? The best minds in silicon valley wake up everyday to automate the jobs of taxi drivers - TFA can be reworded as - 'The misalignment of AI/Tech in Transportation'. Remember the Nepal disaster that happened a couple weeks ago - the largest cranes that they had were stuck in the mud and couldn't move. There were no tools which could help the rescue teams at that time. Its weird that billions have been spent on making a ride automated to make a taxi driver redundant but no improvement in tech for rescue teams.
I know that this comment section is not astroturfed, but it’s really uncanny how different comments are today compared with thread about solving navier stoke
All “fields medalist” signatories - a rarefied and elitist group indeed.
I wish this letter could be more egalitarian and include the view points of those who AREN’T the beneficiaries of a highly competitive winner-take-all system.
Since the common narrative is that AI frees up labor to do other things (engineering -> trades), maybe we can celebrate that genius mathematicians will now spend time teaching children how to be as smart as them?
I was under the impression that mathematics (and science generally) had the primary goal of helping us understand our universe better than those who came before us.
I always wondered how Idiocracy got to the point where they have sophisticated technology and yet everyone is stupid. I think we have our answer.
"We are witnessing a general threat to intellectual work, with misalignment between the outcome of the use of AI and its initial purpose"
Suppose we eventually have GPT-7-class models running practically on $100 devices, with their activity transparent, inspectable, and reproducible. At that point, what exactly is left for us to fear from this threat?
Can't all the prestige-maxed mathematicians still study all these famous problems after ai solves them. And even if they convince open-ai to stop dunking on them, some normal user with gpt 7.1 on the normal chat interface will do it in a year.
Even if they stopped anyone from releasing ai proofs for five whole years it would be meaningless seeing as these problems are decades old already. Humans weren't JUST about to solve them until openai stepped on their toes.
Taking a snapshot of the state of AI math right now and concluding that it will be net negative to human understanding and insight in the future is very short sighted. This statement will be used to promote ideas and actions that will ultimately be disastrous for our country.
At some point we will lose track of all the ai discoveries that are worth remembering.
Academia with the publication system had a way of retrieving old discoveries and build upon them.
If my LLM session found something groundbreaking in between the billion tokens it produced, how would you ever know?
This is really only a short-term problem where the AI companies only have the internal models that can solve these. In the “long” term, which could honestly mean months, everyone will have access to Bel/C/D-level models capable of solving these anyway.
Is the purpose of mathematical research to understand the ‘truth’ of numbers? or be the person who find that truth? I think people who are interested in finding the truth won’t care where it came from.
I'm unsure how any sufficiently advanced AI would not lead to cognitive handoff/the described problems.
chain of credit is important, and plagiarism is harmful.
But is science/mathematics ultimately a pursuit of knowledge, or a pursuit of recognition?
Recognition helps keep people motivated, but that shouldn't be the pursuit of science or mathematics.
Title: Terence Tao - SAIR Competitions and the Future of Experimental Mathematics
https://m.youtube.com/watch?v=rB9YOi3lb7w&pp=ygUSVGVycmVuY2U...
Given the existence of this technology now and the incentives of the AI companies, both of which are not going away; what's a good future here?
A major part of the complaint is that there's no conceptual understanding and building of new ideas coming out of the AI proofs, thus defeating the purpose of the original pursuit.
If in 2027 the AI models start producing, with every mathematics or science breakthrough they make, well-written documents tailored for human understanding, with intermediate concepts, expositions of failed-but-once-promising paths, etc. Would that be good alignment with the mathematics community?
I'm not convinced this is an alignment or technology problem.
If my boss vibe coded an app for the customer and then assigned me to get it working, it would be impossible to maintain. If he gave me enough AI tokens to vibe code the MVP myself and to my design, I wouldn't mind.
I think the same issue is at play here in maths. OpenAI owns the model and they can direct it as they please. They chose to spend lots of money getting a quick result, instead of developing mathematical infrastructure for the next generation of problems. The managers are in charge rather than the experts.
It's a turning point for science and beyond. AI has shown itself to be transformative. Even today, it is already changing the how research in math (and other sciences) is conducted. In the near future, whether it is LLMs or some other superior method, its capabilities are only expected to grow. The time to ask the question is now: Will AI be arguably the best tool at scientist's disposal, or will it instead be paraded around as a super brain collective that no human or group of humans can compete with, discouraging entire new generations of future scientists from ever entering the field? The jury is out on this one.
I am curious what does the first and probably the last human who solved a millennium problem thinks about this.
They can get with the program or be the equivalent of a genius SWE writing assembly on punchcards in 2026.
The only thing I read from this is their ego being bruised by a machine.
If these people cared more about discovery and advancement of human knowledge the only thing they should be doing is celebrating. There's no proof of plagarism but that's an independent issue.
How are they not realizing that in the future children will be able to do impossibly hard math but they will be doing something we can't even think of as of now.
One world class mathematician in the future could be advancing mathematics the equivalent of one Riemann hypothesis A DAY.
How are they not celbrating this as the achievment of the centry? Who cares about plagarism at this scale. It has been solved and it wouldn't have been without AI.
For every benefit that sillycon valley has produced in the recent past, there have been many more harms. I am confident that this will be no different. Of course, benefits and harms depend on one's vantage point.
The main fear seems to be that if mathematics is done at this speed and in this way, humanity will lose its intuition for doing mathematics.
Could we develop new ways to develop understanding and explore new ideas, such as interacting with the models to explain and understand their proofs, as well as to brainstorm related directions to pursue?
As I understand the article, its title should be: "A Severe Misalignment of AI Frontier Companies with Ethics."
Seems more like a misalignment b/w the people practicing mathematics and the people ultimately footing the bill for their work.
Governments are invested in solving mathematical problems for practical purposes. Up to now, achieving these practical purposes relied on mathematicians doing their mathematician thing, which is better defined as a social activity than the achievement of a practical result. Now, governments can achieve similar practical results w/o the need of the social activity.
I don't believe it to be productive to think of the problem wrt AI or AI-company alignment. These conflicts always existed, but they were easy enough to paper over and believe in heavily subsidized fictions that folks in government ever cared about things that mathematicians cared about.
OpenAI should just apologize for having been too greedy. That’s it, as simple as it gets. The fact that it never will is the biggest red flag.
Open source developers have been used by corporations who took their code and created closed SaaS companies.
Now it is the turn of mathematicians who voluntarily contribute ideas, strategies and almost finished proofs in their writings and prompts to closed PaaS (Plagiarism as a Service) companies.
OSS developers have never been respected by the parasites, neither will mathematicians. Your Fields Medals do not protect you from tech bro narcissists. You are a human resource.
>Often these solutions are announced in a rush, leaving no time for a proper writeup, the isolation of new methods and ideas, and citing relevant previous work of others.
That's just capitalism seeping through a previously unexplored crack into academia, and attempting to do the only thing capitalism knows to do - maximize profits - with no additional concern.
I don't know, people. We still really don't know how OpenAI or others are producing these results. It's all very hand wavy and trust-me-bro. How much money/time/compute have they really thrown at these problems? How much human involvement was there? What LLM did they even use? How much regular software was involved? They have given answers to some of those questions but no proof that that's actually what they did. I don't know if it's worth giving them this much credit (which is what we are doing by writing these essays and spending so much time debating). Anthropic wrote a C compiler that turned out to not really be a ready made replacement for GCC. Did they ever do any more work on it? Has anyone else produced a C compiler? It seems like that and these proofs are just demoware that are not (yet? Who knows?) production ready to turn the world upside down. Impressive one-off demos, yes, but companies have been pulling those off for centuries without ever going anywhere afterwards.
I wonder what Demis Hassabis thinks about this. I thought he cared a lot about mathematics.
What changes exactly is this post asking for?
An experiment I would like to see:
Train an LLM with no advanced math texts: only basic math up to 6th grade, conversational text and literary works.
Interact with it (you cannot refer to anything past 6th grade math since you don't know it yourself) and get it to propose a solution to a real world problem. e.g., come up with RSA to practically secure communication.