logoalt Hacker News

jerf • today at 6:26 PM • 1 reply • view on HN

You are asserting that these changes are somehow tied to time, by the very construction of your experiment in that you ask the same questions sequentially day-by-day. However, that leaves you unable to be sure whether the changes are actually tied to the time because it is not being controlled for, as near as I can tell.

As near as I can tell from your "Method and Limitations" from the previous day's run, you are only asking a given AI a given question once. You need to ask the same AI the same question a number of times, preferably as close to simultaneously as possible, to see if there's as much variance within a single day as there is in a week.

As your experiment is constructed, it is not possible for you to collect the data adequate to prove the claim you are trying to make. As I think it is likely that the variance you are seeing is the randomness in the answer process itself and not tied to time at all; two people can probably ask the same question at the same time and get two different answers.

If nothing else, have an AI go back over your experiment, perhaps feed it this comment as well, and ask it what else may be flawed in your experiment. That doesn't mean you have to adopt every suggestion; every experiment can always gather more data and always have an expanded scope and always cost more money. However this is a more-or-less fatal problem with your experiment and there may be others, this is just all I'm considering. An AI can bang on the problem in a more dedicated fashion than I'm doing here.


Replies

mainmailman • today at 6:51 PM

That’s not the main takeaway I had from the article. They asserted that it’s just due to the fact that AI will search a bunch of sources when given a query, but they won’t query the same results each time, so just being more popular means you’re more likely to get a shoutout from the probability machine. Whether or not that 2nd query comes one minute or one day or one month after isn’t as important for what they’re finding.