I’ve had a similar experience in physics. Excellent domain knowledge and semantic search, but the intellectual sparkle and reasoning just isn’t there and it still often throws out a lot of wrong ideas. It is very useful for coding.

But there is a discrepancy between (the implication behind) these reports and how I subjectively feel talking to LLMs. Granted, I don’t have access to whatever cutting edge model is out there for as many credits, but I also don’t feel like I’m talking to an IMO silver medallist.