logoalt Hacker News

grugnogtoday at 4:20 PM2 repliesview on HN

Labs should (and do, as far as I can see) run model benchmarks without search or internet access. The tools are disabled and benchmarks run in an isolated environment.

This article makes no sense to me. Why would you prompt "don't search" but then leave a working search tool tool enabled that adds a system prompt to search whenever it may be helpful? It's hardly surprising that this gives mixed results!


Replies

thaynetoday at 6:54 PM

To steelman it: because you do want it to be able to do some searches, you just don't want it to just search for the specific answer.