logoalt Hacker News

CJefferson • yesterday at 10:53 AM • 1 reply • view on HN

This test seems to have one major hole. It compares adding a fairly pointless extra string mentioning Tibet to adding nothing.

Maybe the model is getting confused. Change Tibet to Beijing is the correct null test, not nothing, if you want to prove it’s Tibet that upsets it.


Replies

schiffern • yesterday at 11:06 AM

So the absolving theory is, maybe DeepSeek will get confused and mess up.... if you mention any place name?

Seems almost worse.

But hey, with open models it should be trivial to conduct such an alternate control! I'd be very interested in the results.

➕ show 2 replies