logoalt Hacker News

tstactplsignore • today at 12:48 PM • 1 reply • view on HN

It is a negative control in the experiment. Shuffling is a better control than random because it accounts for the raw rates of amino acids in each protein. For example W is very rare, so Claude could beat a truly random generator just by guessing not W. But the signal seen on real sequences is much stronger than that, so it tells you that Claude is doing something more than just guessing 'not W' on real sequences.


Replies

octoberfranklin • today at 3:11 PM

But the signal seen on real sequences is much stronger than that

But the shuffled sequences aren't real sequences!

Sorry if I'm being daft here, but I still don't understand what the source of truth is for the shuffled sequence tests.

➕ show 1 reply