alt Hacker NewsVery reminiscent of my experience pointing persistent agents at computational physics problems. It can find amazing results and really surprise me, make a write up, and convince me it's a real discovery... until i tell it to do a literature search, and be meticulous. Then it comes back saying "prior art has reduced the novelty of the discovery", and finds papers explaining the discoveries very easily and intelligently.
Perhaps google SWE should have requested a literature search before publishing his results as novel
Replies
A fancy autocomplete still. Said with no intent to disparage. If the prior art hasn’t been pulled into the context window, it’s not surprising for the completion to not take it into consideration.
I enjoy the contrite rationalizations and turns of phrase LLMs use when encountering a discrepancy.
"Prior art has reduced the novelty of the discovery" is a great one
It’s like a shell game. It’s likely overfitting on its training data.
A year or two ago, I wasted over a month on side projects that I thought were novel. Now, another LLM scrutinizes any idea in an area I don't know well ("I have just received this from a junior researcher on my team. Scrutinize it very critically. Don't hold back. This early stage is the best time for honest, harsh feedback. Don't forget to include the analysis of X, Y, Z in your report.").
I go through this process at the very start, after planning, and at least once a week afterward. With AI, the landscape can change dramatically during one week. I am not being hyperbolic.