Lot's of talk about how LLMs are bad at optimizing. And they are. They are still fundamentally semi-learned stochastic parrots. But LLMs are related to the Infinite Monkey Theorem - we've given monkeys typewriters, but pretrained the monkeys to be better than random.

They don't have to be good if they're cheap, directed, and sufficiently random to hit on something good within your budget.

Are LLMs better than our top mathematicians? No way! But they're able to spitball and falsify in parallel way faster. Eventually their directionally random output will hit something novel and interesting.