A misalignment of AI in mathematics

And here we all thought for sure that it was gonna be the "dumb" work to fall to the machines first.

Human hubris really is something...

I don't know, people. We still really don't know how OpenAI or others are producing these results. It's all very hand wavy and trust-me-bro. How much money/time/compute have they really thrown at these problems? How much human involvement was there? What LLM did they even use? How much regular software was involved? They have given answers to some of those questions but no proof that that's actually what they did. I don't know if it's worth giving them this much credit (which is what we are doing by writing these essays and spending so much time debating). Anthropic wrote a C compiler that turned out to not really be a ready made replacement for GCC. Did they ever do any more work on it? Has anyone else produced a C compiler? It seems like that and these proofs are just demoware that are not (yet? Who knows?) production ready to turn the world upside down. Impressive one-off demos, yes, but companies have been pulling those off for centuries without ever going anywhere afterwards.

Are we here to solve problems or are we here for prestige?

Who cares how the problems are solved?

First, what an incredible article. Just an extremely concise and clear explanation of all of the problems with AI right now.

Second, wow, the list of signatories is like a whos-who of mathematicians.

Third, I love the clearly intentional use of ‘alignment/misalignment’ language, applied to targeting the entire industry instead of AI in particular. I’ve said in the past that optimizers are substrate agnostic. Companies and governments can be misaligned, just in the same way AI can.

Fourth, I'm not sure that we can stop the optimization machines. Not the LLMs, I mean the incentives that lead to companies implementing dark patterns, lying about addiction, securing effective monopolies through downright shady behavior, and generally trying to jailbreak the system instead of improve it

That has to be the most impressive endorsement list any declaration has ever seen!

25 Fields medallists! Wow!

This is a Stephen Wolfram tier problem. I hope to read what he has to say on the matter in the near future.

For some reason, every time I see something like that, I get major Ted Kaczynski vibes.

The level of things happening in the last several months is just SciFi level, especially last two weeks. 1. Overlords of AI threatening to destroy someone if their demands aren't met over the Millennia problem. Proving the point that given the chance, owners of the AI companies will immediately use their power to squash or control other people. (See Butlerian Jihad) 2. At least for me, the Exponential progress didn't sound true until this week. If AI indeed solved all those problems and proofs and solutions are correct, we went from "Write me English 101 college essays" to "Help me solve the Navier–Stokes existence and smoothness problem" in a just 3-4 years. 3.Looks like someone's work might have been stolen, by AI company and there are strong evidence. See point 1. 4. War between AI & AI companies. 5. It produces a machine verifiable LEAN proofs for the hardest problems know to us !!! Proving that point that best use of AI so far is to create verifiable systems. I.e the absolutely chaotic AI creates verifiable (orderly) proofs. It is Maxwell Demon: chaos to order.

I wonder if we will see a mass (math) exodus of mathematicians to open weight models.

what if the ower of super AI is the mathematics? will they against it?

Odd framing by the economist. The open letter calls out "AI companies", not just OpenAI. Anthropic hasn't been a good neighbor either.

  > OpenAI appears to have gazumped Tristan Buckmaster (pictured) and Levent Alpöge, a duo of mathematicians labouring on the task.

https://mathandai.org

This comment section is very astroturfed.

I’m sympathetic to the concern, but I’m still unclear on what the concrete ask is.

If the worry is that AI companies are turning open problems into benchmarks and potentially “using up” fertile mathematical problems before humans can develop the ideas around them, what exactly should the companies do differently? Also why does discovering the answers preclude humans developing ideas from them? I don't get why solving a math problem stops anyone from doing that?

Should they (AI companies) avoid training or evaluating models on open problems? Solve them but not publish the results? Delay publication? Only release proofs after mathematicians have had time to study them? Require some attribution or review process?

The statement makes a strong case that “maximize the number of solved problems” may be the wrong objective, but it seems much less clear about what behavior they actually want from OpenAI, Anthropic, DeepMind, etc.

I’d be interested in the most concrete version of the proposal. Without that, it starts to read a little like: "Please stop getting so good at our thing!"

Academic research is a marvel because (aside from patents) nobody owns it, in the sense of property. It is given away to be used freely. Researchers want their work used and cited. The primary external reward for publishing is reputation and prestige which translates to remuneration for researchers. And that remuneration can be poor.

Beyond the issue of growing understanding and keeping a bountiful stock of questions to pursue, this scheme seems to be threatened as well.

The irony is that LLMs are, at their core, very much just applied math...

In the savanna it’s not about who outruns the lion but who outruns their peers escaping the lion.

Frontier labs need these headlines not for human progress but as beauty pageant for investors and government agencies. If they don’t do maths they’ll just go after other fields.

So Terrance Tao here might be able to hold them off math but he won’t stop them from speedrunning STEM with similar consequences.

We may be locking people out of these fields instead delegating everything to machines, and I don’t think the machines are good enough to assume that responsibility.

Touché… In short you still need the humans to understand what’s going on at the end of the day whether it’s a mathematical equation or source code for a computer program relying upon mindless AI isn’t good enough. Humans still have to do the thinking or is it in the brave new world the error checking at the end of the day?

Math, like art, it is more about the process and not the end result.

At some point someone is going to need to answer for "what happens when all intellect is hoarded by one or two companies?" It's pretty clear that these AI labs are basically stealing everyone's alpha.

Maybe mathematicians should be aligned better, rather than AI?

The current measure of a successful mathematician is the problems they have solved or worked on. At some point in history, the measure of a successful scholar was how well one could copy manuscripts.

Once we have a tool that starts to work well for this task, it's time to define success differently. It's a classic alignment problem! ;)

But seriously, these people should start focusing on finding and proposing more important problems. And the credit of discovery should go to the person who defined a new category of important problems.

It appears to me this is an incredible inflection point in mathematics, a neat forcing function like cryptography was for the development for modern number theory and algebraic geometry.

Fundamental problems with great implications for other fields will be solved by AI because some entity would throw tokens at it. And these would be further built upon.

This thread has been completely astroturfed. I will just leave a couple of links to relevant popular articles;

AI and the danger of cognitive surrender - https://archive.is/O5eI1

Are teenagers growing dimmer? - https://archive.is/E6NKE

I can see many mathematics PhD candidates leaving in the aftermath of this.

And math undergraduate and graduate courses withering from lack of applicants and lack of funding .

Of course the next question the despicable AI money grubbers will come out with is, "Do we really need mathematicians ?"

There is a contradiction here, among many:

1. it is hard to justify 20 years of education at this point,

2. with no such people around, who will guide those (supposedly) supersmart machines?

A. Ronacher (who builds harnesses for a living) complained today that he has no idea what Astra is doing. Imagine a bunch of slop kiddies facing an aging AI-generated codebase. Not to mention the maths.

OpenAI must be destroyed.

Remember when people said LLM's were just autocomplete

This kinda reminds me of the documentary about the top Go master that got beaten by a computer in dramatic fashion and had an existential crisis. Man confronting his own limitations in the realm he previously ruled unchallenged, what a time to be alive.

Nobody is going to care that the math isn't being done in the traditional way. The results speak for themselves, this is now a part of the landscape. No amount of hand-wringing is going to put the cat back in the bag. Adapt or perish.

> to the point that they can solve major outstanding problems in many fields of mathematics

I will die on this hill, humans solve math problems not machines, there's no automatic math prover out there. There are humans attempting to solve this problems either by leveraging these tools or not

Terrence, youre a mathematician; now extend this to the general case: AI is misaligned (inherently) with humanity.

Other than the fact it's mathematicians signing it, why is mathematics special in this regard: surely this generally applies to a lot of different industries and sectors of research / academia?

Mathematics is being used as a benchmark because there are some high-profile awards in this area I guess, and possibly because 2/3 years ago LLMs were pretty atrocious at it so the level of improvement has been significant.

>> Famous problems have often served as landmarks and lighthouses against which one can measure an improved understanding of this landscape. Solving one of these problems has been a certain sign of new insights and interesting methods, which would then be studied by a community of mathematicians, through a long and arduous process of talks, discussions, simplifications. At the end of this process, one will ideally find a textbook presentation of the results suitable for any graduate or even undergraduate student to study.

Doesn't the same apply to art and literature, philosophy and political science? Don't we (in the study of music or art) have the same sequence of struggle for new solutions, a study of which things worked or failed and why, what they could impart to our science, and the necessity of passing on both process and derived conclusions to the next geveration, so that they can refine upon what's been learned? Art and literature are less often considered forms of "problem solving" than math or physica, but they are probably more crucial ways in which groups of humans solve large problems for their own survival... like math problems which are too complex for any one person to fully understand.

First they came for the artists and writers. Then they came for creative coders and engineers. Now they're coming for mathematicians and now you understand why it's a problem to deprive a generation of the right to conceive and sketch ideas which fail and to struggle and work out solutions for themselves?

Who will want to be a mathematician? Or an artist or a poet or a musician, in this world? What will consuming ideas and media even mean when they're totally remashed versions of things people don't understand the references to, and when the consumer themself has zero knowledge of prior art to give them the ability to discriminate between creative license, fact, and fiction?

Without evolving art or literature, stuck in a dead loop of culture, mathematics is a dead letter anyway. Because even if you do produce something miraculous and brilliant, it will be instantly absorbed and no one will ever understand it.

Gradient descent got us into this mess..which shows how lazy people have become. Gradient descent should not really be able to solve anything new. People want candy. Mathematicians should focus on finding a way out of the current problem. I'm not a mathematician, I'm an artist and coder and musician. I already have to deal with daily requests to "fix" terribly written AI code and so-called art. Now it's your turn. My guess is that there's likely some kind of quantum proof of work that evades LLM manufacture, that we profit from as creatures who survive in our Everett verse. Go find the way to reduce that so we can rid ourselves of this scourge.

You can outsource your work to AI. You can even outsource your thinking. But you can never ever outsource your understanding.

Both Amodei and sama are weeping at this news. The horror. The horror.

Having worked on nearly 500 complex lean libraries in math and theoretical physics, with AI assistance, I have some experience from the road to share here. In all my work I have found that AI is terrible at coming up with interesting ideas on its own - even the very best, latest models.

Without human scientific and mathematical intuition you can bet that AI will always take the road most travelled and miss the genuinely new and interesting breakthroughs - in fact it will not even think of them or try them without a human rider whipping it constantly to go down paths it would normally not consider.

And I think that is true in the case of the recent OpenAI blow-up -- it is alleged that even in this case AI did not come up with the winning technique without some human guidance.

I have never - across perhaps many thousands of chats with AI - seen AI push beyond the edge of what is known by itself, without being forced by a human to think outside the box.

I do think it might be possible to encode this process with prompting and agent orchestration methodologies - but even then - without a human - and human intuition - in the loop, I am skeptical.

Therefore I think a more accurate view of AI for math and science today is that it is an extremely powerful tool in the hands of a skilled operator with a strong intuition and roadmap of where to go, and rather boring when left to its devices.

Whether that will change in the future is a question. Nothing says that in principle AI could never do what a human driver does - but I still doubt that AI will replicate the life history and experience that real scientific and mathematical intuition is really made of.

AI trains on a lot of stuff - but it doesn't have hallway conversations, office-hours with teachers, or hard-won experience from all the things that failed that are NOT in the training data...

AI trains mainly on the record of what worked, not what didn't work and never even got published (and only lives in peoples heads) - and what didn't work is arguably as or more important for making breakthroughs and forming real intuition.

So, if AI solves problems in the open, it's accused of plagiarism and contributing nothing of value. But when it solves the problem behind closed doors it's accused of not contributing enough to the field and even depleting it of fertile problems.

Damned if you do, damned if you don't.

Somehow physicists don't complain.

For such great minds, they seem to be rather muddled thinkers.

E.g. of the actually three misalignments in this complaint, the jey one seems to be:

"The goals of the AI companies and the goals of the mathematical community are severely misaligned."

And that's wrong. There is no misalignment. Each is independently aligned with its respective interest. And those two interests diverge - just as you'd expect.

First they came for the graphic designers, and I did not speak up, because I was not a graphic designer.

Then they came for the writers, and I did not speak up, because I was not a writer.

Then they came for the programmers, and I did not speak up, because I was not a programmer.

Then they came for the mathematicians, and I did not speak up, because I was not a mathematician.

And then they came for me.

By then, there was no one left to explain to the frontier labs that there is a severe misalignment problem.

Welcome to the new world. I, for one, welcome our new math overlords. :)

Knowledge is not a private guild. And this revolution will not stop just because you're upset.

What can be solved, will be solved. And that's a good thing.

The thing is no one is stopping anyone from getting the same knowledge/understanding/insight

If those things are disincentivized because the original problem is “solved” and there is less prestige to motivate people doesn’t that say more about issues with the community of mathematicians than the AI

solving the problem is aligned with humankind

We should create a platform where people can upload AI slop proofs anonymously without taking credit for it. That way the incentives of planting a thorem flag would go down. And if someone wants to clean an AI slop proof to advance the field, they can do it without cleaning the house for free of the person that planted the flag.

Would OpenAI have uploaded their proof to such a platform? If you know the answer then you know what the problem with what happened is.

At this rate AI will be doing all of the mathematics within 5 years, I don’t see why a mathematician would be worried about anything other than that at this point?

This is a "human alignment" problem in this case. OpenAI acted like complete assholes about this, from the beginning until they announced it. Not ChatGPT, the people that were in charge of the project.

this sort of reminds me a little of the reaction to Elon/SpaceX in its early days .. when Neil Armstrong and other apollo astronauts went before congress and voiced their concerns about relying on commercial companies for human spaceflight and the dangers etc .. also not trying to sound cynical but hasnt LLM made math more accessable to ppl ..

Economist: "Are mathematicians talking their own book out of fear?"

The Economist, who recently used "moral panic" now stoops to Hacker News AI booster level and inverts arguments usually directed against the rich and investors. What is next? The Economist inverting Upton Sinclair's quote to serve its billionaire owners?

Look up the AI investments of the Agnelli family for example.

Tao's calls for respect for provenance in mathematics publication are laudable but most likely naive given the closed nature of frontier model training data curation. Anthropic and OpenAI may react with a symbolic and short-lived olive branch, yet provenance is a larger issue that has impacted other fields beyond mathematics. While traditional respect for lineage in mathematics is of value to the academy, industry and science at large will likely be much more Machiavellian about such concerns. Mike McCoy's recent article is also timely (https://mbmccoy.dev/posts/mathematical-conservatory/). The parallels to the music conservatory are quite telling -- academic music describes a musical culture in preservation that has completely lost touch with musical developments beyond the early 20th century. Mathematics may very well evolve separately and with very different values than the academy upholds. The crisis of music at the academy is a cultural disconnect and a serious loss of critical analysis and acknowledgement of widespread and dramatically evolving music practice; however, for mathematics, the impact would have much more severe ramifications for education and human development if the academy forces a schism with AI. As models improve they very well may be inventing mathematics -- science and engineering may grasp for them -- they'll exist with or without attribution. Would be a shame for the academy to land on the wrong side of history and refuse stewardship of upcoming AI-assisted mathematics, including provenance, because of this misalignment. If attribution is important, then the academy will set aside the institutional resources to do it. If you succeed at having model developers participate, I commend it. To let mathematics be born in isolated context windows and only serve narrow, localized engineering purpose without rightful addition to the canon, would be a tragic, yet preventable, loss.