The feedback loop of mathematics research
In the wake of OpenAI's announcement of their solution to the Navier-Stokes problem, and the many other announcements of AI-discovered research breakthrough we've been seeing on a weekly — sometimes daily — basis, it's safe to say mathematicians are anxious about what's to come and about what role, if any, will be left for them to play in future mathematics research. Of the many opinions and hot takes being shared, my favorite one was from Scott Aaronson, who described his feelings in these eloquent words:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.
One of the worries is that we as mathematicians are about to become completely redundant. If the machines are better than us at proving things, what are we worth? After all, proving things is the quintessential activity of our profession, right? But I think that point of view is misguided, and I want to explain why I believe that even in a world where AI models are the ones putting the final QED on every new mathematical theorem, human mathematicians are still needed; and to be clear, I actually don't think we are even living in such a world yet.
The feedback loop and its components
Mathematics research is a feedback loop that generates new knowledge in a constant process of re-examining, digesting and distilling existing knowledge and attempting to improve on it. Theorems and proofs are one of the outputs of this feedback loop, and that output goes back in as input to keep feeding the feedback loop and keep it going. But, crucially, theorems and proofs are neither the only type of output of the loop, nor sufficient as input to keep the loop going productively.
The AI-generated math breakthroughs we are hearing about were produced by AI models that have been taught the existing cumulative body of knowledge, almost entirely created by humans at this point, up to the cutoff point in time for that model's training. So I will describe the new knowledge that they are generating as being "distance one away" from human-generated knowledge.
Let's imagine that tomorrow morning an AI model is released that completely beats humans in the ability to prove new theorems when starting from a starting point of properly distilled mathematics knowledge (of the sort that the current corpus of human-created knowledge represents). The models currently being tested internally at OpenAI and Anthropic may actually be close to having those characteristics. Such a model will in short order generate all new theorems and their proofs that are distance one away from the corpus of knowledge it is given as the starting point for its investigation. Human mathematicians won't stand a chance in competing with it in doing that, and may be inclined to despair that their careers are over.
But guess what? Life does not end with those new theorems. A proper feedback loop would keep building new knowledge on top of that new knowledge that was generated, proceeding to uncover beautiful new mathematical ideas that are at distance two, three, four, and so on from that initial body of knowledge. But left to its own devices, the superhuman AI theorem-prover will not be able to do that. With AI running autonomously, the feedback loop will grind to a halt shortly after finishing the distance one theorems.
What will the AI be missing? Something that I believe it doesn't currently come even close to having: the ability to reflect on the knowledge it generated, simplify it, identify and extract the essential components of what makes a new proof or method work, identify interesting connections of a new result to other results and areas, and keep generalizing the ideas to the point where they can be used across many different kinds of situations. These are the other, completely essential (but mostly overlooked in the current discourse) elements of the feedback loop of mathematics research.
The current AI models are incredible at generating new distance one theorems, but seem to have very little ability to explain or synthesize the new knowledge they are generating. Their proof writeups are perceived as slop and annoy a lot of human mathematicians; they are certainly no fun to read for humans even when they report amazing breakthroughs. The thing is, I'm pretty sure they are not fun for AIs to read as well! In my experience, AIs can appreciate well-presented content and insights, and can make use of them, to a similar extent as human mathematicians do. What this means is that if OpenAI or some other company tries to completely hijack the research feedback loop, what I predict they will discover is that after an initial wave of successes, their models will start drowning under the weight of their own mountains of self-generated slop, and their progress will halt.
By contrast, if humans are kept in the loop we are going to see some incredible progress, with humans using AI to generate new knowledge, then adding their own insights and ability to distill, communicate, and generalize the knowledge in the ways that currently only humans know how to do, then using that distilled knowledge to generate further new knowledge with the help of AI, keeping the feedback loop going in a very efficient and productive way. With this collaborative process, it won't even always be clear who is doing the "proving", the human or the AI, since there isn't always a clear boundary delineating what counts as "proving" from what counts as "improvements in our understanding" or "partial progress towards a proof".
Proxies and tools
In the recent open letter "A Severe Misalignment of AI in Mathematics", 28 Fields Medalists expressed the view that "solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight". This resonated with many people in the community, and I've seen similar sentiments shared in various places. I'd like to point out a nuance that I think hasn't been addressed in this discussion, and that my analysis above tries to pinpoint. It's in fact quite reasonable to regard "solving problems" as a goal in its own right; there's nothing dishonorable in my view about wanting to know whether a statement like "zeta(5) is irrational" is true and getting excited about reading a proof of such a breakthrough result and finally having that question answered in a convincing way, independently of any insights or conceptual understanding that the proof may also help us achieve. Actually, different people care about "conceptual understanding and insight" to different extents, and not everyone is so philosophically high-minded as to claim that those lofty concepts are the only things that really matter. Personally, I am sometimes turned off by over-philosophical editorializing in papers I'm reading, and just want to see the damn proof. (I'd even guess that the writers of the "severe misalignment" letter, being famous mathematicians who have solved some of the most difficult problems known to mankind, must themselves have felt the exhilaration that comes with solving those very difficult problems, and that that exhilaration must have been at least partly in due to the "mere" act of solving the problems — the "proxy" — and not just to the satisfaction of having created new insights.)
The point I am making is that, even if we consider solving problems to be a worthy goal, independently of the understanding that is reached in the process of solving those problems, the AI companies would still be misguided if they were to try to hijack the research feedback loop, for the simple reason that they won't actually succeed in doing so except in a short-term sense of eating up all the "distance one" research. In other words, one does not need to argue that theorems and proofs aren't really that important, or that they are only "a tool and a proxy" for philosophically more important things, to reach some of the conclusions that the authors of the Severe Misalignment letter are reaching.
Reasons for optimism
Obviously it seems impossible to predict how things will play out and how long it will be before AI models catch up to human mathematicians not just in their reasoning and proving abilities but also in their ability to communicate, distill and simplify knowledge in a way that is essential to keeping the feedback loop of mathematics research going. For the time being, humans still have a role to play, and I see it as quite an exciting and valuable one.
Written without AI using ReelDocs
Reel (proof of human authorship): https://reeldocs.io/d/SwG4Egu8hDy8e4